|2.||Test System Design|
The surge in use and popularity of graphical user interfaces (GUIs) over the last 10 years has been attributed to both technological enhancement (the availability of higher quality monitors which render it possible to produce distinct, detailed images on the screen) and theoretical assumptions e.g., a suitable graphical representation is easier to comprehend (Lodding, 1983) and direct manipulation affords a greater sense of user control (Shneiderman, 1992). Specifically, experiments on human memory for pictures or the effects of imagery on memory (e.g., Mandl and Levin, 1989) lend support to the broad notion of iconic representation in graphical interfaces enhancing usability compared to text-based interfaces. Similarly the control of interfaces afforded by a mouse or equivalent input device is considered to be optimal with respect to human information processing (Card et al, 1983).
Despite the increased use of GUIs in IR applications and Web search engine interfaces, it is not clear that such interface features enhance usability in all situations. With respect to icons for example, Lansdale et al. (1989, 1990) showed that any gains in speed of the user's visual scanning process could be at the cost of a greater likelihood of missing the target. Other research suggests that it is less the iconic representation than the associated use of text labels on icons that aids users in GUIs. (For a good review of such work see Rogers, 1989).
One area where icons in combination with direct manipulation may offer significant advantages over text input is in terms of selecting objects on a screen. Here, established models of human performance in target selection (e.g., Card et al (1983) Model Human Processor and GOMS analysis framework) could be used to demonstrate quantitatively any speed or efficiency advantage. However, in IR applications, searching is less a matter of hitting targets on a screen than formulating queries that are conceptually appropriate, and in such cases it is not at all appropriate to recast the task in terms of target selection and mouse movement. As a result, it is not logically the case that the use of GUIs will enhance searcher performance.
Drawing up reliable and valid recommendations for interface features for any task is complicated further by the possible interaction of user type on performance with particular interface styles. Chui and Dillon (1997) report data showing that users experienced with animation in graphical interfaces are more likely to be affected by its presence or absence than users with no experience of animation. In information retrieval tasks, it has long been known that there exist large individual differences in performance (see e.g., Fenichel, 1981) and it is possible that visual components of search interfaces might be an important source of this variance, or may effect novice user performance with an IR system. Allen (1994) demonstrated an interaction between logical reasoning ability and interface style in an information retrieval application, and Sein et al (1993) showed that users' scores on a test of visual ability served as a predictor of learning success on a graphical interface. Thus, individual differences among users seem to be a significant source of variance in determining the usability of all IR interface components.
In the present paper we report a usability evaluation of a prototype IR interface aimed ultimately at supporting novice user access to library materials on art. While the study fed into the early stages of the design process of a new tool, we were mainly interested in seeing how two key properties of a graphical interface (the use of icons and the capability to select options with a mouse) might improve novice searcher performance and perceived satisfaction with the tool.
The major differences between the interfaces from the users' point of view were the ability to point and click (enabled in the graphical) instead of typing commands (required in the textual interface), and the presence of images (as opposed to text) supporting the categorisation of the database contents in the graphical interface.
The data in the system mainly consist of reference materials in the fields of art and art history. The initial design change was aimed at providing two levels of iconic representation above the final output of relevant resources and their locations (which was identical for both interfaces). The data structure adapted for both interfaces was hierarchical. At the root node, the categories for classifying the data are based on broad concepts such as name of artist, picture, and scripture. Each category was broken down to subcategories. Icons were designed to represent the underlying object or function to which they refer. In order to maximize the effectiveness of graphical representation, pictorial icons were drawn by an artist, one of the design team members. The system does not allow users to do keyword searching but supports browsing only. Apart from the interfaces both databases were identical in content and structure.
It should be noted here that the user type variable manipulated expertise in searching with IR systems, not domain (art) expertise. The domain variable is likely to be worthy of further investigation but is not central to the focus of this system design.
Subjects were required to answer a set of 10 questions (see Appendix). These questions were developed by the authors to ensure that the questions did not unduly favor any one interface and that users would need to explore the database fully to find the answers.
After five tasks, subjects took a short break and then proceeded as before on the second interface. Order of presentation was counterbalanced across all subjects with half the experts and half of the novices starting on the textual interface, and the other half of both user groups starting on the iconic interface. Order of questions was randomized for each subject (there were no a priori reasons for allocating certain questions to specific interface conditions).
While no main effect was observed for interface style (F[1,22] = 1.19, p > .05), by examining the time data in full (see Table 1), we can see that experts performed approximately 15% faster, and novice users about 12% faster, with the graphical interface. In usability terms, such differences could be important (see e.g., Landauer, 1991). There was no interaction effect.
Differences between novices and experts approached significance (F[1,22] =3.07, p<.07) with experts performing slightly more accurately than novices. A general trend favoring the graphical interface is apparent but is not statistically significant. It should be noted that 4 novice users gave up searching with the textual interface (one each on question 1 and question 6, and two gave up on question 3) and one novice user gave up the search with the graphical interface (question 2). All experts completed all tasks. Scoring accuracy only on a 3-point scale with high rates of non-completion introduces possible ceiling and cellar effects and thus may have served to constrain variance in these data.
Analysis indicated a significant effect for interface style (F[1, 22] = 4.606, p < .05).
Tukey's HSD follow-up test for interface styles revealed that the number of nodes visited by both user groups were significantly higher with the textual interface than with the graphical interface (p < .05). These data are important as they suggest that users were able to gain more direct access to relevant information when icons were employed (see Table 3). For novices this translates approximately into a 15% efficiency gain in navigation, for experts, 10%.
Both user groups' protocols indicated a tendency to get lost more easily on the textual interface as demonstrated in the navigation measures reported above. Novice users in particular had many difficulties in navigating the textual interface. As noted earlier, four novices gave up answering certain questions.
Obviously the task type may be a major source of the variance in user response to interface style, and searches involving less abstract and more textual material , e.g., title or author searches, may not be so affected by graphical interfaces at the search system. This is one area for further research. But in the present context, the effect on navigation in a comparatively small database is likely to be magnified as the application grows to include many more entries. Thus, the use of the graphical interface is likely to enhance user performance for typical users of the full database in the library.
1. Find examples of the printmaker Dore's work.
2. Locate several books with maps of Chicago.
3. Find a US geological map of Mississippi.
4. Find a current map of Asia
5. Find a source for multiple pictures of different butterflies.
6. Find several examples of Impressionist painting?
7. Find one book with illustrations by Picasso.
8. Find examples of Afro-American artists.
9. I need examples of colonial American architecture. What book do I have to refer to?
10. Where can I find pictures by the photographer Eliot Porter?
Allen, B. (1994). Cognitive abilities and information system usability Information Processing and Management, 30, 177-191.
Card, S., Moran, T. and Newell, A. (1983). The Psychology of Human Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
Chui, M. and Dillon, A. (1997). Who's zooming whom? Attunement at the human-computer interface. Journal of the American Society for Information Science (in press).
Fenichel, C. H. (1981). Online searching: Measures that discriminate among users with different types of experience. Journal of the American Society for Information Science, 32, 23-32.
Landauer, T. (1991). Let's get real: a position paper on the role of cognitive psychology in the design of humanly useful and usable products. In J. Carroll (ed.) Designing Interaction. Cambridge MA: Cambridge University Press. 60-73.
Lansdale, M.W., Jones, M.R. and Jones, M.A. (1989). Visual Search in Iconic and Verbal Interfaces. In E. Megaw (ed.) Contemporary Ergonomics, 1989. London: Taylor and Francis. 422-429.
Lansdale, M.W., Simpson, M. and Stroud T.R.M. (1990). A comparison of words and icons as external memory aids in an information retrieval task. Behaviour & Information Technology, 9(2), 111-131.
Lodding, K. N. (1983). Iconic interfacing. IEEE Computer Graphics and Applications, 24, 11-20.
Mandl, H. and Levin. J. (1989). (eds) Knowledge Acquisition from Text and Pictures. Amsterdam: Elsevier Science Publishers. 3-35.
Meadow, C.T., Wang, J. and Yuan, W. (1995). A study of user performance and attitudes with information retrieval interfaces. Journal of the American Society for Information Science, 46, 490-505.
Norman, K. (1991). The Psychology of Menu Selection Norwood, NJ: Ablex.
Rogers, Y. (1989). Icon design for the user interface. International Review of Ergonomics, 2, 159-164.
Sein, M. Olfmann, L., Bostrom, R. and Davis, S. (1993). Visualization ability as a predictor of user learning success. International Journal of Man-Machine Studies, 39(4), 599-620.
Shneiderman, B. (1992). Designing the User Interface, 2nd Edition. Boston, MA: Addison Wesley.