Evaluating The Effectiveness Of BEN In Localizing Different Types Of Software Fault

Abstract

Debugging refers to the activity of locating software faults in a program and is considered to be one of the most challenging tasks during software development. Automated fault localization tools have been developed to reduce the amount of effort and time software developers have to spend on debugging. In this thesis, we evaluate the effectiveness of a fault localization tool called BEN in locating different types of software faults. Assuming that combinatorial testing has been performed on the subject program, BEN leverages the result obtained from combinatorial testing to perform fault localization. Our evaluation focuses on how the following three properties of software fault affect the effectiveness of BEN: (1) Accessibility: Accessibility refers to the degree of difficulty to reach (and execute) a fault during a program execution; (2) Input-value sensitivity: A fault is input-value sensitive if the execution of the fault triggers a failure only for some input values but not for other input values; and (3) Control-flow sensitivity: A fault is control-flow sensitive if the execution of the fault triggers a failure while inducing a change of control flow in the program execution.We conducted our experiments on seven programs from the Siemens suite and two real-life programs, grep and gzip, from the SIR repository. Our results indicate that BEN is very effective in locating faults of low accessibility. A unique aspect of BEN is that it generates a failed test called the core member and a small group of passed tests called the derived members. The spectrum of the core member is compared to that of each derived member. The way in which the core and derived members are generated allows faults of low accessibility to be effectively located by BEN. Our results also suggest that in most of the cases, BEN is effective in locating input value- and control flow-insensitive faults. However, no conclusion can be drawn from the experimental data about the individual impact of input value sensitivity and control flow sensitivity on BEN’s effectiveness.