Summer REU: Verification and Validation for Software Testing

Sample Projects

An introduction tutorial on software safety will be provided at the beginning of the summer program. It covers the basic definitions and concepts of software safety, techniques for hazard identification and analysis (e.g., FTA – Fault Tree Analysis), design for safety via hazard elimination, reduction, and control, etc. A series of training seminars will also be provided to help less experienced students establish necessary background knowledge and improve their technical and research skills before working on assigned research projects. Additional seminars by experts from Raytheon Network Centric Systems, Lockheed Martin Aeronautics Company, and EDS, an HP Company will be scheduled as well to help the students better understand how software safety is verified and validated in practice for real-life applications. Follow-up meetings will be organized for students to discuss with invited speakers how to address the gap between the current "state-of-practice" and "state-of-art" with respect to software safety.

Listed below are the descriptions of two sample projects.

1) Foundational Requirements for Competency in Software Safety

For software-dependent systems in which failures can result in significant property damage, injury, or death, software safety must be a principal consideration. Software engineers involved with such projects in both the government and private industry require an understanding of existing software safety standards and practices, as well as an awareness of emerging trends and developments in the area. To facilitate this, we will create a set of instructional materials appropriate for software engineers no matter what their previous level of experience with software safety.

There are four major parts. The first is to survey existing procedures and techniques, including software hazard analysis, fault tree analysis (FTA), failure modes and effects analysis (FMEA). Approaches to estimate the cost of including safety in software requirements will also be examined. The second is to review and analyze software-related accidents. Examples are the crash of a Korean jet (1997) and an American Airlines jet (1995), explosion of the Ariane 5 launcher (1996), loss of the Mars Climate Orbiter (1999), destruction of the Mars Polar Lander (2000), and the power outage across the Northeastern U.S. and Canada (2003). The role of software in these accidents will be examined, and ways to avoid these problems in the future will be suggested. The third is to understand the limitations and problems of software safety standards such as NASA-STD-8719.13A, Military Standard MIL-STD-882D, DO-178B, and others. We will document lessons learned from these standards as a repository of the "state of the art." We also report pragmatic issues of applying such standards in practice. The fourth is to apply the instructional materials as the basis for a series of training seminars to prepare engineers at our industry partners, including Raytheon Network Centric Systems and the Lockheed Martin Aeronautics Company. The feedback on the training will be used to help further improve the materials.

2) Testing for Software Safety

Testing for consistency between implementation and functional specifications does not provide safety assurance. It is difficult to generate tests for safety testing by only using the hazard analysis results presented in the fault tree models, because of the lack of an explicit and common description of the relationship between a fault tree and the corresponding functional specifications and safety requirements. We will develop an efficient method for test generation using both functional specifications and safety requirements. In addition to testing software under normal conditions, tests so generated can also verify whether the implementation has properly handled all the hazardous conditions identified by the fault tree analysis on safety requirements.

We need to examine not only the relationship between the inputs and outputs of a software system but also that between the inputs and the impact of the outputs on system behavior. Traditional testing in general is success oriented. The focus is on testing what software should do; less emphasis is placed on what it is not supposed to do or on the effects of failures and errors on safety-related behavior. This implies that invalid inputs are often not used and responses to failures or omission errors are seldom tested. As a result, even if software behaves according to its functional specifications, it can still be unsafe. To overcome this problem, software should be tested against the safety constraints and safety-related functional requirements derived from a hazard analysis. Although hazard analysis by itself cannot ensure safety, it is a necessary step before hazards can be eliminated or properly controlled. In other words, we can make a system design safer via appropriate hazard analysis and subsequent testing of the design to ensure that it meets the required level of safety. One significant challenge in performing such analysis is selecting a good model that matches the project’s goals, tasks, and skills. In this research, we propose to use fault tree analysis (FTA), which has been widely used as a way to analyze causes of hazards.

Our research objectives are twofold: to integrate functional specifications with fault tree models for testable safety analysis and to generate safety tests from the integrated specifications. The first objective allows potential components and system failures to be specified explicitly together with the intended behaviors of a system. The second objective aims at detection of potential failures through systematic safety-driven testing. We note that this research focuses on testing whether or not the hazardous conditions identified by design-level fault tree analysis will occur in the target implementation. Functional specifications often focus on intended behaviors of a system and are intrinsically incomplete. If complete interactions between hazardous conditions and intended behaviors are specified, they can be used to generate safety tests. Research along this line remains to be seen, however, due to the heterogeneity of functional specifications and fault models (e.g., statecharts versus fault trees). A critical problem is that some basic events in a fault tree may have no counterpart in the corresponding functional specifications.

The proposed approach consists of three major parts.

a)	Integration of fault tree models into functional specifications so as to identify testable interactions between intended behaviors and hazardous conditions.
b)	A test generator that produces not only functional tests but also safety tests for a target implementation. The generated safety tests cover all the failure paths in a given fault tree model.
c)	A testing environment for executing generated functional and safety tests and evaluating test results against expected behaviors or hazardous conditions. It includes a test harness as well as an environment simulation of external events and conditions.