HCI Bibliography Home | HCI Journals | About JUS | Journal Info | JUS Journal Volumes | Detailed Records | RefWorks | EndNote | Hide Abstracts
JUS Tables of Contents: 010203040506070809

Journal of Usability Studies 5

Editors:Avi Parush
Dates:2009/2010
Volume:5
Publisher:Usability Professionals' Association
Standard No:ISSN 1931-3357
Papers:17
Links:Journal Home Page | Table of Contents
  1. JUS 2009 Volume 5 Issue 1
  2. JUS 2010 Volume 5 Issue 2
  3. JUS 2010 Volume 5 Issue 3
  4. JUS 2010 Volume 5 Issue 4

JUS 2009 Volume 5 Issue 1

Invited Editorial

Engaged Scholars, Thoughtful Practitioners: The Interdependence of Academics and Practitioners in User-Centered Design and Usability BIB 1-7
  Susan M. Dray

Peer-reviewed Articles

When Links Change: How Additions and Deletions of Single Navigation Links Affect User Performance BIBAHTMLPDF 8-20
  Lauren F. V. Scharff; Philip Kortum
This study examined user performance for Web sites in which a critical navigation link may have changed over the course of two visits. These second visits occurred either immediately after the first visit, one week later, or three weeks later. A shortcut link to the information was either consistently present, consistently absent, or varied between user visits. Results indicated when a pertinent navigation link was removed users searched through more pages and were less accurate in finding target information. When a link was added after the first visit, only about half of the users used it; however, even users who didn't use the added link still showed a performance improvement in their subsequent visit, suggesting that completing a single previous search on a site will lead to relatively long-term memories that can influence subsequent searches.
   The following were the main findings of the study:
  • By the second time users visit a site, their performance improves compared to
       their first visit (decreased page counts and search times), even after
       delays of up to three weeks.
  • Even small changes, like the addition or deletion of a single link, can have
       large impacts on user performance.
  • Presence of a shortcut link to vital information dramatically increases the
       probability that a user will find that information. Compared to a site
       without the link, users will be faster and visit fewer pages.
  • When a pertinent navigation link is removed, users search through more pages
       and are less accurate in finding target information.
  • When users have clear memory of a site, deletion of a link causes users to
       exhaustively examine the primary navigation structure; this behavior
       persists as the delay between visits increases for at least up to three
       weeks.
  • When a link is added after the first visit, only about half of the users will
       use it. The rest will continue to use the site in the way they have done in
       the past.
  • Even users who don't use an added link still show a performance improvement
       in their subsequent visits, suggesting that secondary paths to information
       should not be deleted simply because a top-level navigation link has been
       added.
  • Even if the link is removed after a user has been exposed to it, users are no
       less accurate than if the link had never been present.
  • The Combined Walkthrough: Measuring Behavioral, Affective, and Cognitive Information in Usability Testing BIBAHTMLPDF 21-33
      Timo Partala; Riitta Kangaskorte
    This paper describes an experiment in studying users' behavior, emotions, and cognitive processes in single usability testing sessions using an experimental method called the combined walkthrough. The users' behavior was studied using task times and completion rates, and emotions were studied using bipolar scales for experienced valence and arousal. Cognition was studied after each task by revisiting detected usability problems together with the users and applying an interactive method based on cognitive walkthrough to each usability problem. An interactive media application was tested with 16 participants using these methods. The results of the experiment showed that the developed methods were efficient in identifying usability problems and measuring the different aspects of interaction, which enabled the researchers to obtain a more multifaceted view of the users' interaction with the system and the nature of the problems encountered.
       The following were the main findings of this experiment:
  • Behavioral, affective, and cognitive aspects of computer system usage can be
       cost-effectively studied together in usability testing.
  • The information obtained by the behavioral, affective, and cognitive
       measurements can contribute to a more multifaceted understanding of user
       interaction with the system.
  • Variations in the users' emotional experiences (valence and arousal) related
       to completing a task using an interactive system can be efficiently measured
       using bipolar scales. Systematic measurement of emotional experiences
       broadens the scope of subjective measures beyond traditional satisfaction
       measures.
  • The use of highly positive or negative media elements influences overall
       ratings of task-related affective experiences in interactive media
       applications.
  • Ideas underlying the cognitive walkthrough can be useful in retrospective
       analysis of usability problems together with the user.
  • How To Specify the Participant Group Size for Usability Studies: A Practitioner's Guide BIBAHTMLPDF 34-45
      Ritch Macefield
    Using secondary literature, this article helps practitioners to specify the participant group size for a usability study. It is also designed to help practitioners understand and articulate the basis, risks, and implications associated with any specification. It is designed to be accessible to the typical practitioner and covers those study scenarios that are common in commercial contexts.
       The following were the main findings in this article:
  • For a wide variety of reasons, specification of the participant group size
       for a usability study remains a matter of opinion and debate.
  • The goal for usability practitioners is to help negotiate a group size that
       is optimal, or at least beneficial, for the wider project in which the study
       is taking place. This means that practitioners should be able to articulate
       the basis, risks, and implications associated with any specification.
  • When utilizing research literature in this area, practitioners should
       carefully consider how well the specific studies underpinning the particular
       research relates to the study under consideration. Similarly, they should
       pay careful attention to any caveats in the advice being offered.
  • There is no "one size fits all" solution to the challenge here. However, for
       studies related to problem discovery a group size of 3-20 participants is
       typically valid, with 5-10 participants being a sensible baseline range. In
       these scenarios, the group size should typically be increased along with the
       study's complexity and the criticality of its context. In scenarios
       concerned with discovering severe ("show stopping") problems in early
       conceptual prototypes a group size of five participants is typically valid.
       For comparative studies where statically significant findings are being
       sought, a group size of 8-25 participants is typically valid, with 10-12
       participants being a sensible baseline range.
  • In many scenarios, it can be beneficial to split study groups into chunks of
       participants within a punctuated study, whereby the results data is
       incrementally analyzed after each chunk. One benefit of this tactic is that
       a study can be terminated early if its objectives have already been met,
       thereby saving project resources. For example, a comparative study may be
       terminated early because it has already produced the statistically
       significant findings being sought. Another benefit of this tactic is that it
       promotes the iterative design processes that are fundamental to a UCD
       philosophy. For example, in a study of an early conceptual prototype, one
       chunk of participants revealed a show stopping problem. After the interface
       design was revised, the study continued.
  • JUS 2010 Volume 5 Issue 2

    Invited Editorial

    Is Technology Becoming More Usable -- or Less -- and With What Consequences? BIBHTMLPDF 46-49
      Daryle Gardner-Bonneau

    Peer-reviewed Articles

    Reliability of Self-Reported Awareness Measures Based on Eye Tracking BIBAHTMLPDF 50-64
      William Albert; Donna Tedesco
    Participants in a usability evaluation are often asked whether they noticed certain elements after some level of interaction with a design. The purpose of this study was to evaluate the reliability of self-reported awareness measures using eye tracking data. Participants were shown 20 popular homepages for 7 seconds each and then asked afterwards if they saw 2 particular elements on each page. The results showed that self-reported awareness measures are reliable, but can vary depending on question structure and object type. These findings have implications for how usability practitioners ask questions about object awareness, and how that information is used in the design process.
       Practitioner's Take Away
  • Usability practitioners should feel confident in collecting self-reported
       awareness measures from participants.
  • If a practitioner wants to minimize the chance of making an incorrect
       conclusion, they should use a continuous (5- or 7-point) scale for
       self-reported awareness (similar to Experiment 2).
  • If a practitioner wants to maximize the likelihood of confirming that a
       participant did or did not see an element, they should use a discrete set of
       questions for self reported awareness (similar to Experiment 1).
  • Participants are more reliable in their self-reported awareness for
       navigation and image elements, than functional elements, regardless of
       question structure.
  • Usability Evaluation of Randomized Keypad BIBAHTMLPDF 65-75
      Young Sam Ryu; Do Hyong Koh; Brad L. Aday; Xavier A. Gutierrez; John D. Platt
    In this work, the usability of a randomized numeric keypad was examined and compared to the usability of a conventional numeric keypad. The comparison used completion time measurements and the error rate of short (4 digit) and long (8-digit) PINs to contrast efficiency and accuracy of the keypads. The results showed that the average completion time with a randomized keypad is longer than with a conventional keypad. Additionally, the number of errors with a randomized keypad was significantly higher than with a conventional keypad, particularly when using long PINs. Accordingly, a randomized numeric keypad is more applicable to tasks with short (4-digit) PINs.
       Practitioner's Take Away
  • Longer completion times should be expected when using a randomized numeric
       keypad versus a conventional keypad.
  • The number of errors with a randomized keypad was significantly higher than
       with a conventional keypad when users typed longer PINs.
  • The number of errors with a randomized keypad was not significantly higher
       than with a conventional keypad when users typed short PINs.
  • A randomized numeric keypad is better suited to applications requiring short
       PINs.
  • JUS 2010 Volume 5 Issue 3

    Invited Editorial

    The Seven Habits of Highly Effective Usability People BIBPDF 76-80
      Mary Beth Rettger

    Peer-reviewed Articles

    Plain Language Makes a Difference When People Vote BIBAHTMLPDF 81-103
      Janice (Ginny) Redish; Dana Chisnell; Sharon Laskowski; Svetlana Lowry
    The authors report on a study in which 45 U.S. citizens in three geographic areas and over a range of ages and education levels voted on two ballots that differed only in the wording and presentation of the language on the ballots.
       The study sought to answer three questions:
  • Do voters vote more accurately on a ballot with plain language instructions
       than on a ballot with traditional instructions?
  • Do voters recognize the difference in language between the two ballots?
  • Do voters prefer one ballot over the other? In addition to voting on the two
       ballots study participants commented on pages from the two ballots indicated their preference page-by-page and overall.
       For this study, the answer to all three questions was "yes." Participants performed better with the plain language ballot. Their comments showed that they recognized plain language. They overwhelmingly preferred the plain language ballot.
       The authors also discuss issues that arose on both ballots from problems with straight-party voting, with mistaking one contest for another, and with reviewing votes. Based on the study results, the authors provide guidance on language to use on ballots. This article includes links to the two ballots, other materials used in the study, and the full report with more details.
       The following are key points from this study:
  • Language matters. Study participants voted more accurately on the ballot with
       plain language than on the ballot with traditional language.
  • Education matters. Level of education correlated with accuracy. Voters with
       less education made more errors.
  • Location, gender, age, and voting experience do not matter. None of those
       factors was a statistically significant correlate of accuracy.
  • People recognize plain language. After they voted both ballots, participants
       were shown pairs of pages (the A and B versions of the same ballot page) and
       were told, "Notice that the instructions on these pages are different.
       Please compare them and comment on them." Participants commented that
       certain words were "simpler," "more common," and "easier to understand."
  • People prefer plain language. Asked for an overall preference between the two
       ballots, 82% (37 of 45) chose Ballot B, the plain language ballot.
  • Straight-party voting confuses many people. Even on the plain language
       ballot, participants made errors related to straight-party voting.
  • Some voters do not have a good grasp of levels of government. Many of the
       errors on both ballots related to confusing U.S. Senate with State Senator
       and County Commission with City Council.
  • Usability professionals can help make ballots and other voting materials more
       usable through research and consulting.
  • Even in a summative test, usability specialists often see ways to improve the
       product for its next release. In the study reported here, the plain language
       ballot did significantly better than the ballot with traditional language.
       Nonetheless, after watching participants work with the ballot, we realized
       we could make the language even clearer. We include recommendations for an
       even better plain language ballot.
  • Response Interpolation and Scale Sensitivity: Evidence Against 5-Point Scales BIBAHTMLPDF 104-110
      Kraig Finstad
    A series of usability tests was run on two enterprise software applications, followed by verbal administration of the System Usability Scale. The original instrument with its 5-point Likert items was presented, as well as an alternate version modified with 7-point Likert items. Participants in the 5-point scale condition were more likely than those presented with the 7-point scale to interpolate, i.e., attempt a response between two discrete values presented to them. In an applied setting, this implied that electronic radio-button style survey tools using 5-point items might not be accurately measuring participant responses. This finding supported the conclusion that 7-point Likert items provide a more accurate measure of a participant's true evaluation and are more appropriate for electronically-distributed and otherwise unsupervised usability questionnaires.
       The following are the main findings of this study:
  • Five-point Likert scales are more likely than 7-point scales to elicit
       interpolations in usability inventories.
  • Interpolations are problematic because they cannot be mitigated within an
       electronic survey medium and require interpretation with facilitated
       surveys.
  • Interpolations provide evidence that 5-point Likert scales may not be
       sensitive enough to record a usability test participant's true evaluation of
       a system.
  • Seven-point Likert scales appear to be sensitive enough to record a more
       accurate evaluation of an interface while remaining relatively compact.
  • Seven-point Likert scales appear to be more suited to electronic distribution
       of usability inventories.
  • Practitioners can quickly test Likert items through verbal protocols by using
       interpolations as a metric.
  • A Comparison of the Usability of a Laptop, Communicator, and Handheld Computer BIBAHTMLPDF 111-123
      Piia Suomalainen; Leena Korpinen; Rauno Pääkkönen
    The goal of this study was to examine the usability of a laptop, communicator, and handheld computer using test subjects and questionnaires. The study aimed to determine how user-friendly and ergonomically correct these devices are. The subjects (25) had 5 minutes to perform typing or calculation tests with each device. While the subjects performed the tasks, an observer monitored the subjects' work posture. After the tasks were completed, the subjects completed questionnaires about the usability of each device Based on the subjects' experiences, the handheld computer and laptop had better ergonomic characteristics than the communicator. Subjects felt the highest amounts of stress in their neck while working on the laptop, subjects felt stress on their backs while working on the communicator, and they felt stress in their eyes while working on the handheld computer. Subjects performed the typing tasks best using the laptop. Our research suggests that companies developing mobile devices should consider ergonomic issues and the ergonomic differences between different types of mobile devices to further improve user satisfaction.
       The following were the main findings of this study:
  • Usability of laptops, communicators, and handheld computers can be determined
       by combining observations of subjects' work posture, tests on how well
       subjects complete tasks, questionnaires on subjects' perceptions of
       ergonomic design, and questionnaires on how subjects' physically felt while
       using the devices.
  • Most stress caused by these devices can be felt in a person's back, upper
       limbs, and eyes.
  • Commentary

    A Commentary of "How To Specify the Participant Group Size for Usability Studies: A Practitioner's Guide" by R. Macefield BIBPDF 124-128
      Rolf Molich

    JUS 2010 Volume 5 Issue 4

    Invited Editorial

    Moving Towards an All-Encompassing Universal Design Approach in ICT BIBPDF 129-131
      André Liem; Sarah J. Swierenga; Rama Gheerawo

    Peer-reviewed Articles

    Comparing Computer Versus Human Data Collection Methods for Public Usability Evaluations of a Tactile-Audio Display BIBAHTMLPDF 132-146
      Maria Karam; Carmen Branje; John-Patrick Udo; Frank Russo; Deborah I. Fels
    We present a public usability study that provides preliminary results on the effectiveness of a universally designed system that conveys music and other sounds into tactile sensations. The system was displayed at a public science museum as part of a larger multimedia exhibit aimed at presenting a youths' perspective on global warming and the environment. We compare two approaches to gathering user feedback about the system in a study that we conducted to assess user responses to the inclusion of a tactile display within the larger audio-visual exhibit; in one version, a human researcher administered the study and in the other version a touch screen computer was used to obtain responses. Both approaches were used to explore the public's basic understanding of the tactile display within the context of the larger exhibit.
       The two methods yielded very similar responses from participants; however, our comparison of the two techniques revealed that there were subtle differences overall. In this paper, we compare the two study techniques for their value in providing access to public usability data for assessing universally designed interactive systems. We present both sets of results, with a cost benefit analysis of using each in the context of public usability tests for universal design.
       We have found that it is important to consider the following concepts when creating systems using universal design principles:
  • Use automated questionnaires to increase participant numbers in evaluations.
  • Use public usability studies to supplement lab experiments with real-world
       data.
  • Modify existing evaluation tools to extend into the public domain.
  • Include as wide an audience in evaluations to ensure universality of the
       design.
  • Expect to alter major features of the design to ensure target users are
       addressed.
  • Select only technology that is robust enough to withstand constant public
       use.
  • Reinforce and secure systems to ensure safety of users and the equipment.
  • Restrict public studies to short, succinct questions and questionnaires to
       maintain ease of ethics approval, and focus the study on the broader aspects
       of system interactions.
  • Ensure proper assistance is in place to accommodate users with special needs
       during the study. Sign language or other interpreters, or special access
       needs are essential to address when conducting studies on universal designs.
  • Online Learning: Designing for All Users BIBAHTMLPDF 147-156
      Cindy Poore-Pariseau
    During the fall of 2008, 4.6 million students pursued their education in online environments in the United States (Allen & Seaman, 2009). Considering that students with disabilities represent nearly 10% of all U.S. college students (National Council on Disability as reported by Frieden, 2003), one can see a need to disseminate information regarding how to best meet the needs of this population as they look to further their education by taking advantage of online learning opportunities. Through this paper, the reader will learn about Universal Design for Learning (UDL), accessibility laws, how the laws affect online education, and how instructional design can be implemented as a way to increase access to education for college students with disabilities. Finally, there will be an exploration of the impact accessibility laws have on instructional design and how an increase in accessibility can improve motivation for all segments of the population.
       Note: Although the statistics cited are based on U.S. postsecondary education student populations, the ideas outlined herein can be applied beyond the U.S., as the needs of disabled students are universal.
       Practitioner's Take Away
       As the number of students choosing to complete their studies in online environments continues to grow, so will the number of disabled students enrolling in online courses continue to grow. The following points must be considered when an online course is being developed:
  • Recognition must be given to the fact that retrofitting accommodations in
       online environments is not only an arduous task, it is often impractical,
       necessitating training in concepts such as Universal Design for Learning to
       be placed at the forefront.
  • Importance must be placed on ensuring that instructional designers (including
       faculty members who design their own instruction) receive the training
       necessary to become familiar with disabilities and disability related laws.
  • Importance must be placed on ensuring that instructional designers (including
       faculty members who design their own instruction) receive the training
       necessary to develop the ability to resolve accessibility issues (or to
       partner in resolving such issues).
  • A conscious effort must be made to proactively incorporate accessibility
       standards into all course rooms, coursework, and course materials, so that
       all have comparable opportunities to contribute effectively to the
       educational process.
  • All involved parties, including society as a whole, will benefit from a
       product that is fully usable and accessible.
  • Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility BIBAHTMLPDF 157-171
      Panayiotis Koutsabasis; Evangelos Vlachogiannis; Jenny S. Darzentas
    The current set of tools and specifications for ensuring web accessibility require expert knowledge and often have a highly technical orientation, with the consequence that it is not very clear how, or even when, to make use of them. In an attempt to tackle this problem, this paper reviews the types of tools and specifications available and proposes a simple and practical methodology for web accessibility evaluation that demonstrates how these tools and specifications could be used. The proposed methodology proposes methods and processes for reaching and maintaining web accessibility, and consists of the following phases: (a) identification of user requirements and setting up of accessibility goals, (b) web accessibility evaluation and redesign process, and (c) establishment and follow-up of accessibility policy. Further, in order to illustrate step (b), an example of web accessibility evaluation is described, where the domain is contemporary scientific publishing web sites. The work presented in this paper reports on issues that need to be considered by human-computer interaction (HCI) researchers, interaction design practitioners, and usability professionals for inclusive web design and are complementary to web usability engineering.
       The following are the key points of this paper:
  • Web accessibility concerns every user, designer, and business owner. There
       are many ethical, business, user, and technical arguments for designing for
       web accessibility.
  • Current web accessibility tools and specifications are technically oriented
       and need expert knowledge to be understood and applied.
  • Despite the work on web accessibility, most web sites are still inaccessible.
       For example (described in this paper) six out of ten e-publishing homepages
       pose major accessibility problems and only two out of ten are accessible.
  • Typical web accessibility problems found include the loss of (a) information
       organisation, (b) navigation, (c) visibility, and (d) user control, when
       users access web sites in constrained contexts (e.g., with an oral browser
       or in a keyboard only situation).
  • The paper proposes a practical methodology reaching and maintaining web
       accessibility, which includes (a) identification of user requirements and
       set up of accessibility goals, (b) web accessibility evaluation and redesign
       process, and (c) establishment and follow-up of accessibility policy. This
       three step approach will help to achieve the following:
  • Identify particular user requirements when designing for web accessibility
       and conform to a level of accessibility specifications (e.g., single-A of
       WCAG 1.0).
  • Implement a fast and practical method for regular web accessibility
       inspection according to your accessibility goals. This paper illustrates a
       method that makes use of automated tools and heuristics that can be employed
       in accessibility inspections.
  • Understand the appropriate time and the value of user testing. User testing
       is important for web accessibility, but a basic level of accessibility
       should be there. Testing with disabled users also increases web developers'
       awareness.
  • Establish and follow up a web accessibility policy. This should focus on
       basic rules for content update of the web site that editors of the web site
       should follow as well as tools for checking content on the fly or in the
       background. This is an area for further research development and application
       in the field.
  • Intra- and Inter-Cultural Usability in Computer-Supported Collaboration BIBAHTMLPDF 172-197
      Ravi Vatrapu; Dan Suthers
    In this paper, we argue for an increased scope of universal design to encompass usability and accessibility for not only users with physical disabilities but also for users from different cultures. Towards this end, we present an empirical evaluation of cultural usability in computer-supported collaboration. The premise of this research is that perception and appropriation of socio-technical affordances vary across cultures. In an experimental study with a computer-supported collaborative learning environment, pairs of participants from similar and different cultures (American-American, American-Chinese, and Chinese-Chinese) appropriated affordances and produced technological intersubjectivity. Cultural usability was analyzed through the use of performance and satisfaction measures. The results show a systemic variation in efficiency, effectiveness, and satisfaction between the two cultural groups. Implications of these findings for the research and practice of usability, in general, and cultural usability, in particular, are discussed in this paper.
       The following are the key points of this paper:
  • Demand characteristics, in the context of cultural usability, refer to the
       study of expectations, evaluator-participant relationships, and cultural
       norms for appraisal and attribution. Demand characteristics should be given
       careful consideration in the design of cultural usability evaluations and in
       the interpretation of the results.
  • For usability evaluations involving participants from social collectivist
       cultures with interactional concerns for deference and harmony-maintenance,
       it might be beneficial to use questionnaires that solicit both Likert-type
       ratings and open-ended comments.
  • Participants from highly collectivistic cultures (such as the Chinese
       participants of this experimental study) with a high-context communication
       style and a field-dependent perceptual style might offer higher overall
       ratings to a system despite offering lower subjective ratings for the
       constituent parts of the systems.
  • Participants from highly individualistic cultures (such as the American
       participants of this experimental study) with a low-context communication
       style and a field-independent perceptual style might offer lower overall
       ratings to a system despite offering higher subjective ratings for the
       constituent parts of the systems.
  • Participants from highly collectivistic cultures with a greater emphasis on
       deference and harmony-maintenance (the Chinese participants in the context
       of this study) might make a higher number of positive comments compared to
       negative comments during usability evaluation despite higher negative
       ratings on Likert-type questionnaires.
  • Participants from highly individualistic cultures with a lesser emphasis on
       deference and harmony-maintenance (the American participants in the context
       of this study) might make fewer positive comments and more negative comments
       during usability evaluation despite higher positive ratings on Likert-type
       questionnaires.
  • Efficacy of open-ended comments might be higher for inter-cultural user
       groups when compared to intra-cultural user groups.
  • Despite significant differences in objective measures of usability, there
       might be no significant differences in performance or achievement of the
       primary task. In other words, there is a possibility of a "performance
       preference paradox" in the usability evaluation of computer-supported
       collaboration systems.
  • It might be more productive and informative to conceive of cultural variation
       at the level of human-computer interaction as a variation in the perception
       and appropriation of action-taking possibilities and meaning-making
       opportunities relate to actor competencies and system capabilities and the
       differences in social relationships and expectations rather than homeostatic
       cultural dimensional models or stereotypical typologies.
  • Design the technological capabilities of the system taking into account the
       competencies of the users. Design should focus on what action-taking
       possibilities and meaning-making opportunities users can psychologically
       perceive and sociologically appropriate. Design for sustaining,
       transforming, and creating traditional and novel configurations of social
       relationships and interactions in socio-technical systems.