Combinatorial Testing in Cloud Computing (eBook)

Wei-Tek Tsai is currently a professor at both the School of Computing Informatics and Decision Systems Engineering at Arizona State University, USA and the School of Computer Science and Engineering at Beihang University, China. He received his PhD and MS in Computer Science from the University of California at Berkeley, and his BS in Computer Science and engineering from MIT. He has produced over 400 papers in various journals and conferences, received two Best Paper awards, and awarded several Guest Professorships. His work has been supported by the US Department of Defense, Department of Education, National Science Foundation, the EU, and industrial companies such as Intel, Fujitsu and Guidant. In the last ten years, he has focused his energies on service-oriented computing and SaaS, as well as various aspects of software engineering including blockchain, architecture, testing, and maintenance.Guanqiu Qi received his PhD in Computer Science from Arizona State University in 2014. His research interests span many aspects of software engineering, such as SaaS (Software-as-a-Service), TaaS (Testing-as-a-Service), Big Data Testing, Combinatorial Testing, and Service-Oriented Computing, as well as blockchain.

Preface 6
Contents 8
1 Introduction 12
1.1 Software Testing 12
1.2 Cloud Testing 13
1.3 Combinatorial Designs 14
1.3.1 Latin Square 14
1.3.2 Orthogonal Array 14
1.3.3 Covering Array 15
1.4 Combinatorial Testing 16
1.4.1 Covering Array for Testing 17
1.4.2 Automatic Efficient Test Generator 17
1.4.3 In-Parameter-Order 19
1.4.4 Genetic Algorithm 19
1.4.5 Backtracking Algorithm 20
1.4.6 Fault Detection 20
1.5 Structure of This Book 21
References 22
2 Combinatorial Testing in Cloud Computing 25
2.1 Combinatorial Testing in Cloud Computing 25
2.2 Improvements of Combinatorial Testing in Cloud Environment 26
2.3 Faulty Location Analysis in Combinatorial Testing 27
2.3.1 Fault Localization Based on Failure-Inducing Combinations 27
2.3.2 Identifying Failure-Inducing Combinations in a Combinatorial Test Set 28
2.3.3 Faulty Interaction Identification via Constraint Solving and Optimization 29
2.3.4 Characterizing Failure-Causing Parameter Interactions by Adaptive Testing 30
2.3.5 Comparisons of Existing Faulty Location Analysis Solutions 32
2.4 Related Work 32
References 33
3 Adaptive Fault Detection In Multi-tenancy Saas Systems 34
3.1 Adaptive Testing Framework 34
3.1.1 Learning from Earlier Test Results 34
3.1.2 AR Algorithm Framework 37
3.1.3 Relationship Between Faults, Final-Faults, and Candidate-Faults 38
3.2 Simulation of AR Algorithm 38
3.3 Incremental Testing to Allow New Components 41
References 45
4 Test Algebra for Concurrent Combinatorial Testing 46
4.1 Test Algebra 46
4.1.1 Learning from Previous Test Results 47
4.1.2 Changing Test Result Status 49
4.1.3 Matrix Representation 49
4.1.4 Relationship Between Configuration and Its Interactions 51
4.1.5 Merging Concurrent Testing Results 52
4.1.6 Distributive Rule 54
4.1.7 Incremental Development 54
4.2 Conclusion 55
A Commutativity of otimes 46
B Associativity of otimes 55
C Commutativity of oplus 56
D Associativity of oplus 57
E Distributivity of otimes Over oplus 59
Reference 61
5 Concurrent Test Algebra Execution with Combinatorial Testing 62
5.1 TA Analysis Framework 62
5.1.1 The Role of N in Concurrent Combinatorial Testing 67
5.1.2 Modified Testing Process 67
5.2 TA Analysis Algorithm 68
5.3 TA Analysis Process and Related Considerations 69
5.3.1 Analysis Process 69
5.3.2 Adjustment in Analyzing 70
5.4 Test Database Design 71
5.4.1 X and F Table Design 71
5.4.2 P Table Design 71
5.4.3 N and U Table Design 72
5.5 Experiment 72
5.6 Conclusion 76
References 76
6 Test Algebra Execution in a Cloud Environment 77
6.1 TA Concurrent Execution and Analysis 77
6.1.1 TA Concurrent Execution 77
6.1.2 NU Configuration 79
6.1.3 NU Configuration Selection Algorithms 80
6.1.4 Analysis Process of NU and U Configurations 82
6.2 TA Experiments 84
6.2.1 TA MapReduce Experiment Flowchart 84
6.2.2 Different Configuration Numbers of TA Experiments 84
6.2.3 Different Speedup Strategy for TA Experiments 84
6.2.4 Different Fault Rates for TA Experiments 85
6.2.5 Explanation on Simulated Data 87
6.2.6 Simulation with Different Clusters 87
6.2.7 Simulation using 37-node Cluster with Different Map Slots 87
6.3 Conclusion 89
Reference 90
7 Adaptive Reasoning Algorithm with Automated Test Cases Generation and Test Algebra in Saas System 91
7.1 Experimentation Using a MTA SaaS Sample 91
7.2 SaaS Testing 94
7.3 SaaS Test Case Generation 96
7.4 Simulation and Analysis 100
7.4.1 Simulation of ARP Algorithm 100
7.4.2 Incremental Testing with Automatic Test Generation 102
7.4.3 Simulation Experiments of ARP+PTR 103
7.4.4 Analysis of the Strategy on Test Generation 104
7.4.5 TA Simulation in SaaS 107
References 107
8 TaaS Design for Combinatorial Testing 108
8.1 TaaS Introduction 108
8.2 TaaS Design with TA and AR 109
8.3 TaaS as SaaS 112
8.3.1 GUIs 114
8.3.2 Workflows 114
8.3.3 Services 115
8.3.4 Runtime Composition, Execution and Scalability 115
8.4 Experimental Results 117
8.5 Conclusion 119
References 119
9 Integrated Taas with Fault Detection and Test Algebra 121
9.1 Framework 121
9.1.1 Integrated Process 121
9.1.2 Framework Illustration 122
9.2 Experiments and Results 125
9.2.1 Experiment Setup 125
9.2.2 Experiment Results 129
9.2.3 Measurements 129
9.3 Conclusion 133
References 134