Combinatorial Robustness Testing based on Error-Constraints

Robustness is an important property of a software, which must be tested in addition to a software's functionality. This requires invalid values and invalid value combinations to be able to observe a software's reaction to them. Combinatorial testing (CT) is an effective black box test method.
But its effectiveness deteriorates in the presence of invalid values or invalid value combinations.
This phenomenon is called invalid input masking effect and led to extensions of CT that we call combinatorial robustness testing (CRT). The objective of CRT is to improve the fault detection by avoiding invalid input masking.
This is achieved by separating the testing of valid values and valid value combinations from the testing of invalid values and invalid value combinations.
The objective of this work is to develop a new CRT test method with a modeling approach to specify invalid values and invalid value combinations equally well.
This modeling approach should also be incorporated into explicit test adequate criteria and test selection strategies. Furthermore, the CRT test method shall be supported by automated techniques.
Therefore, all concepts and techniques are operationalized and integrated in a test automation framework which includes a process, an architecture, and a Java-based reference implementation.