HCI Bibliography Home | HCI Conferences | ESP Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
ESP Tables of Contents: 868791939697

Empirical Studies of Programmers 1986-06-05

Fullname:Empirical Studies of Programmers
Editors:Elliot Soloway; Sitharama Iyengar
Location:Washington, DC
Dates:1986-Jun-05 to 1985-Jun-06
Publisher:Ablex Publishing Corporation
Standard No:ISBN 0-89391-388-X (cloth) 0-89391-463-0 (ppk); QA 76.6 W688; hcibib: ESP86
Empirical Studies of Programmers: The Territory, Paths, and Destinations BIBA 1-12
  Ben Shneiderman
This paper attempts to describe the varied intellectual territory that programmers work in. It offers several paths for researchers who wish to explore this territory: controlled experiments, observational or field studies, surveys, and cognitive theories. Finally, this paper suggests several important destinations for researchers: refining the use of current languages, improving present and future languages, developing special purpose languages, and improving tools and methods.
Comprehension Differences in Debugging by Skilled and Novice Programmers BIBA 13-27
  Leo Gugerty; Gary M. Olson
Two experiments investigated expert-novice differences in debugging computer programs. Subjects used a microcomputer to debug programs provided to them. The programs were in LOGO in Experiment 1 and Pascal in Experiment 2. We found that experts debugged more quickly and successfully than novices, largely because they generated high quality hypotheses on the basis of less study of the code. Further, novices frequently added bugs to the program while trying to find the original one. We also described some of the debugging strategies the subjects used. At least in these simple programs, experts' superior debugging performance seemed to be due primarily to their superior ability to comprehend the program.
Plans in Programming: Definition, Demonstration, and Development BIBA 28-47
  Robert S. Rist
Support for the use of plans in cognitive models of programs was provided by evidence from both novice and expert programmers. For novice programmers, an initial plan-based description of code segments was replaced by syntactic and control based groups as the programs became more complex. An increase in plan use with expertise was also evident. Experts used only plan groupings in their efforts to understand a program.
   Cluster analysis of these code groupings showed the precise definition and order of appearance of the program plans, providing an experimental basis for identifying cognitive plans. Three main sources of plan emergence were identified in the novice data: goal based, object based and basic plans. Goal based plans allow the focal segment for a goal or program to be easily identified. The link between the program goal and the code that implements this goal provides the basic plan structure of the program. The rest of the program code supports this basic operation. The conceptual model of the program for experts is centered on this focal segment. This directs attention in the understanding and construction of programs. It gives a human solution to the problems of goal search and selection.
Processes in Computer Program Comprehension BIBA 48-57
  Susan Wiedenbeck
Beacons are key features in a program which serve as typical indicators of the presence of a particular structure or operation. According to Brooks's top-down theory of program comprehension, programmers do not study a program line-by-line but rather search for beacons to verify their hypotheses about a program's function. This paper reports on an experiment which used a memorization and recall method to study beacons in program comprehension. Novices and experienced programmers memorized and recalled a short Pascal program. The results showed that experienced programmers recalled key lines, or beacons, much better than other parts of the program. However, novices did not recall them better. This supports the idea that certain key parts do exist in programs, and they serve as focal points for comprehension by experienced programmers.
Cognitive Processes in Program Comprehension BIBA 58-79
  Stanley Letovsky
This paper reports on an empirical study of the cognitive processes involved in program comprehension. Verbal protocols were gathered from professional programmers as they were engaged in a program understanding task. Based on analysis of these protocols, several types of interesting cognitive events were identified. These include asking questions and conjecturing facts about the code. We describe these event types, and use them to derive a computational model of the programmers' mental processes.
Mental Models and Software Maintenance BIBA 80-98
  David C. Littman; Jeannine Pinto; Stanley Letovsky; Elliot Soloway
Understanding how a program is constructed and how it functions are important parts of the task of maintaining or enhancing a computer program. We have analyzed videotaped protocols of experienced programmers as they enhanced a personnel database program. Our analysis suggests that there are two strategies for program understanding, the systematic strategy and the as-needed strategy. The programmer using the systematic strategy traces data flow and control flow through the program in order to understand global program behavior. The programmer using the as-needed strategy focuses on local program behavior in order to localize study of the program. Our empirical data show that there is a strong relationship between using a systematic approach to acquire knowledge about the program and modifying the program successfully. Programmers who used the systematic approach to study the program constructed successful modifications; programmers who used the as-needed approach failed to construct successful modifications. Programmers who used the systematic strategy gathered knowledge about the causal interactions of the program's functional components. Programmers who used the as-needed strategy did not gather such causal knowledge and therefore failed to detect interactions among components of the program.
Design Activity in Developing Modules for Complex Software BIBA 99-116
  A. F. Norcio; L. J. Chmura
Since 1978, the goal of the Software Cost Reduction (SCR) project has been to demonstrate the effectiveness of certain software engineering techniques for developing complex software. The application is the redevelopment of the operational flight program for the A-7E aircraft. Also since then, the Software Technology Evaluation (STE) project has been monitoring SCR project activity in order to provide an objective evaluation of the SCR methodology. SCR project activity data are collected from SCR personnel on a weekly basis. Over 55000 hours of SCR design, code, test, and other activity data have been captured and recorded in a computer data base. Analyses of SCR module design data show that there are parameters that can be used to characterize and predict design progress. One example is the ratio between cumulative design discussing and cumulative design creating activities. This ratio is referred to as the Progress Indicator Ratio (PIR) and seems to be an accurate metric for design completeness. This and other results suggest that discussion activity among software engineers may play a major role in the software design process and may be a leading indicator of design activity progress.
Different Logo Teaching Styles: Do They Really Matter BIBA 117-127
  Rachelle S. Heller
This study has investigated whether a structured Logo experience as outlined by the Minnesota Educational Computing Consortium (MECC) and an experiential Logo setting as proposed by Seymour Papert of Massachusetts Institute of Technology (MIT) affected the level of Logo learning and the conceptual tempo of fourth grade students. Fifty-five fourth grade students from a private parochial school in suburban Washington, D.C. were divided into the two Logo learning groups. Each group met for one hour of Logo experiences each week for twelve weeks. All subjects took a Logo syntax test after three weeks and after twelve weeks of study. All subjects also took a Logo content test, a Logo concept test, a pre and post-test of conceptual tempo (MFFT: Child), and answered an opinion questionnaire after twelve weeks of study. The data from these tests were analyzed using a t-test. The attitude questionnaire data were analyzed using Chi-square (X2) and Kendall's tau. A journal was used to record the weekly meetings.
   The results revealed that the students in the structured environment scored higher on the test of Logo content. In areas of Logo syntax and allied concepts to Logo, there was no statistical difference between the groups. Furthermore, both groups improved over time in their understanding of Logo syntax as measured by the syntax test. There was a statistically significant improvement for the subjects in the Papert group as measured by their error rate on the MFFT: Child. The results from the attitude questionnaire, while not statistically significant, suggested that the subjects in a Papert environment like to work alone at the computer.
   This study suggests that if the goal of exposure to Logo is a thorough knowledge of Logo, then it is expedient to use a structured approach. The study suggests that the educational community examine the goal of Logo in the classroom.
Programmer/Nonprogrammer Differences in Specifying Procedures to People and Computers BIBA 128-137
  Lisa A. Onorato; Roger W. Schvaneveldt
This paper investigates the effects of computer programming skills on the writing of ordinary instructions. Three computer experience groups (Naive, Beginner, Expert) wrote telephone directory instructions to one of three targets (another person, George Washington, or an English-understanding computer). Each subject performed this task twice, such that data could be collected from Beginners before and after one semester of computer experience. No effects of this Session variable were found, however, leading to a discussion of the learning versus selection hypotheses.
   Other results indicate that overall type of instruction strategy used was only significant for the computer target. When writing for a computer, Experts were more likely to use a whole name search, Naive users to use a separate name search, and Beginners to use a letter search. These results are interpreted in terms of problem representation and in terms of expectations of a natural language computer system.
   In addition, it was found that Experts used more looping and programming action statements than Naive or Beginner users, even when writing to another person. The programmer's advantages are then discussed. Also, the computer target was always the least likely to receive information in terms of alphabetization instructions, directory descriptors and fillers suggesting further burdens and expectations that users may place on natural language systems.
The Effects of Program-Dependent and Program-Independent Deletions on Software Cloze Tests BIBA 138-152
  Mark Thomas; Stuart Zweben
The cloze procedure has proven to be a reliable measure of prose comprehension and preliminary evidence suggests it could be a powerful tool for measuring software comprehensibility as well. Early work with the cloze procedure in the software domain failed to take into account some of the factors which can influence the procedure's results. Research by Hall shed light on several of these issues, and suggested that, by varying the proportion of cloze items for which program dependent knowledge is required, the outcome of a cloze test can be altered. This research further examines Hall's findings. His hypothesis that controlling these factors can determine the outcome of the test is confirmed through replication of earlier work by Cook which did not take them into account. Methods for resolving the potentially varying conclusions of these cloze tests are discussed.
Experimental Evaluation of Program Quality Using External Metrics BIBA 153-167
  Fred G. Harold
An experiment is described in which software metrics applied primarily to the source listings of COBOL programs were used to evaluate program quality. The programs analyzed were written by students exposed to structured programming concepts (experimental group) and students oriented to good programming style independent of structured programming (control group). Program quality was measured in terms of readability, maintainability, and modifiability. The metrics were applied by evaluators unaware of the experimental design. Statistical measures (frequency distributions, t-tests, discriminant analysis, and correlation coefficients) were generated from the metric data and used to distinguish the quality incorporated in the programs developed by the two groups.
An Empirical Study of the Effects of Modularity on Program Modifiability BIBA 168-186
  Timothy D. Korson; Vijay K. Vaishnavi
An empirical study of the effects of modularity on adaptive program maintenance is reported. A discussion of methodological issues in empirical studies using programmers is also included.
   The study provides strong evidence that a modular program is faster to modify than a non-modular, but otherwise equivalent version of the same program, when one of the following conditions hold:
  • (a) Modularity has been used to implement "information hiding" which localizes
        changes required by a modification.
  • (b) Existing modules in a program perform useful generic operations, some of
        which can be used in implementing a modification.
  • (c) A significant understanding of, and changes to, the existing code are
        required for performing a modification. In contrast, the study provides evidence that modifications not fitting into the above categories are unaided by the presence of modularity in the source code.
  • Experiments on Slicing-Based Debugging Aids BIBA 187-197
      Mark Weiser; Jim Lyle
    Programming slicing is a method for reducing the amount of code looked at when debugging or understanding programs. Previous work concentrated on showing that programmers mentally slice during debugging. We present new work which concentrates on evaluating automatic tools for presenting slices to the debugging programmer. For one such tool, an online window-based editor/compiler/slicing system, we were unable to show that slicing helped. A second experiment, pencil and paper this time, presented programmers with dices of programs. A dice is a slice on incorrect variables from which slices on correct variables have been removed. Programmers using the dicing tool debugged their programs significantly faster than unaided programmers.
    A Model of Novice Debugging in LISP BIBA 198-212
      Claudius M. Kessler; John R. Anderson
    This paper reports an investigation of novice programmers trying to debug one-line LISP functions. We present a model of debugging based on protocol data and introduce a production system simulation of the ideal novice debugger. We conclude with a discussion of the applicability of such a model to the teaching of programming in LISP.
    Fragile Knowledge and Neglected Strategies in Novice Programmers BIBA 213-229
      D. N. Perkins; Fay Martin
    Many students have great difficulty mastering the basics of programming. Inadequate knowledge, neglect of general problem-solving strategies, or both might explain their troubles. We report a series of clinical interviews of students taking first year BASIC in which an experimenter interacted with students as they worked, systematically providing help as needed in a progression from general strategic prompts to particular advice. The results indicate a substantial problem of "fragile knowledge" in novices -- knowledge that is partial, hard to access, and often misused. The results also show that general strategic prompts often resolve these difficulties. Recommendations for teaching more robust knowledge and general strategies are made. Implications for the impact of programming on general cognitive skills are considered.
    Analyzing the High Frequency Bugs in Novice Programs BIBA 230-251
      James C. Spohrer; Elliot Soloway
    In this paper, we provide a detailed analysis of the bugs that novice programmers most frequently made while solving a set of introductory programming problems. First, we show the special status of high frequency bugs: Lots of students learning to program make the same bugs. Second, we show that most of the high frequency bugs do not arise because students have a misconception about some language construct. The implications of these two results for teaching programming are discussed.
    A Plan for Empirical Studies of Programmers BIBA 252-255
      Victor R. Basili
    For any area which requires experimental evidence, a high-level long range plan is needed. The plan should (1) classify the problems that need to be addressed, (2) provide for the development of appropriate theories and models of explanation, (3) offer a research framework that permits experiments to be categorized and evaluated, and (4) provide a feedback mechanism so that knowledge can be incorporated into the knowledge base of the research and user communities.
    By the Way, Did Anyone Study Any Real Programmers? BIBA 256-262
      Bill Curtis
    The relevance of the current empirical research on programming to the pressing problems of software development is challenged. A review of research since 1980 shows a trend toward greater methodological rigor. However, at the same time most studies concentrate on novice programming, and fail to offer guidance in developing advanced software development environments. Several crucial questions are posed for future empirical research on programming and two exploratory studies under way in the author's laboratory are described.
    What to Do Next: Meeting the Challenge of Programming-in-the-Large BIB 263-268
      Elliot Soloway