System Verification (eBook)

Front Cover 1
System Verification 4
Copyright Page 5
Table of Contents 6
LIST OF ILLUSTRATIONS 16
LIST OF TABLES 20
PREFACE 22
ACKNOWLEDGMENTS 24
LIST OF ABBREVIATIONS 26
Part 1 SETTING THE STAGE 30
Chapter 1.1 The Global Verification Situation 32
1.1.1 The Meaning of the Word Verification 32
1.1.2 Verification Classes 33
1.1.2.1 Item Qualification 34
1.1.2.2 Item Acceptance 35
1.1.2.3 System Test and Evaluation 35
1.1.3 Feedback into Product Models 36
1.1.4 Technical Data Assessment 36
1.1.5 Process Verification 36
1.1.6 Program Assembly of the Verification Process 37
1.1.6.1 High-Rate Production Program 37
1.1.6.2 Low-Volume, High-Dollar Production Program 37
1.1.6.3 One-of-a-Kind Production Program 37
1.1.7 Verification Documentation Intensity 38
1.1.8 In the Aggregate 40
Chapter 1.2 Introduction to System Development 42
1.2.1 What Is a System? 42
1.2.2 System Development 43
1.2.3 Three Steps on the Way to Great Systems 43
1.2.4 Organizational Structure 44
1.2.5 The Systems Approach 46
1.2.6 The Two Vs 48
1.2.7 The Foundation of System Engineering 50
1.2.8 System Development Phasing Overview 53
1.2.9 Toward a Standard Process 56
1.2.10 Development Environments 58
1.2.10.1 The Waterfall Development Model 59
1.2.10.2 The Spiral Development Model 59
1.2.10.3 The V Development Model 61
1.2.10.4 The N Development Model 62
1.2.10.5 Development Environment Integration 63
Chapter 1.3 Requirements Analysis Overview 68
1.3.1 Requirements 68
1.3.2 The Need and Its Initial Expansion Using Traditional Structured Analysis 72
1.3.3 Structured Decomposition Using Traditional Structured Analysis 76
1.3.3.1 Functional Analysis 79
1.3.3.2 Performance Requirements Analysis 82
1.3.3.3 Design Constraints Analysis 84
1.3.3.3.1 Interface Requirements Analysis 84
1.3.3.3.2 Environmental Requirements Analysis 85
1.3.3.3.3 Specialty Engineering Requirements Analysis 88
1.3.4 Computer Software Approaches 88
1.3.5 Verification Requirements 91
1.3.6 Applicable Documents 92
1.3.7 Process Requirements Analysis 93
Part 2 ITEM QUALIFICATION VERIFICATION 96
Chapter 2.1 Verification Requirements 98
2.1.1 Verification Documentation 98
2.1.2 Item Planning Fundamentals 100
2.1.2.1 Traceability Matrix 101
2.1.2.2 Verification Methods 103
2.1.2.3 Product and Verification Levels 105
2.1.2.4 Verification Classes 106
2.1.2.5 Items Subject to Qualification and Acceptance 108
2.1.2.6 Verification Directionality 109
2.1.2.7 Product Verification Layering 109
2.1.2.8 Verification Requirements Definition Timing 110
2.1.3 Verification Requirements Analysis 110
2.1.3.1 Selecting the Method 110
2.1.3.2 Writing Responsibility and Support 111
2.1.3.3 Writing the Verification Paragraph 111
2.1.4 Verification Planning, Data Capture, and Documentation 113
2.1.5 Section 4 Structure 116
2.1.5.1 MIL-STD-961E Structure 116
2.1.5.2 An Alternate Structure 118
2.1.5.3 External Verification Requirements Documentation 119
2.1.6 Verification Computer Databases 119
Chapter 2.2 Top-Down Verification Planning 122
2.2.1 A Matter of Scale 122
2.2.2 Expansion of Function F44 123
2.2.3 Item Qualification Process 125
2.2.4 The Planning Transform 126
2.2.5 Qualification Process Implementation 133
2.2.6 Specific Task Processes 137
2.2.6.1 Generic Test Task 137
2.2.6.2 Generic Analysis Task 137
2.2.6.3 Generic Examination Task 138
2.2.6.4 Generic Demonstration Task 138
2.2.7 Program Task Structuring 138
Chapter 2.3 Item Qualification Test Planning Analysis 140
2.3.1 Overview 140
2.3.2 Qualification Test Program Planning 141
2.3.2.1 Test Philosophy 141
2.3.2.2 Top-Down and Bottom-Up Planning 142
2.3.2.3 Qualification Test Scheduling 143
2.3.2.4 Qualification Test Item Selection 144
2.3.2.5 Item to Task Transformation 146
2.3.2.6 Verification Information Organization 147
2.3.3 Item Planning Cycle 150
2.3.3.1 Verification Matrix Test Data Fusion 150
2.3.3.2 Qualification Process Design 153
2.3.3.2.1 Test Task Goals 153
2.3.3.2.2 Process Analysis 153
2.3.3.2.3 Resource Identification 155
2.3.3.3 Test Procedure Development 156
2.3.3.4 Validate the Plan 158
2.3.4 Integrated Verification Plan Test Integration 159
2.3.5 Special Problems in Verification through Test 162
2.3.5.1 Life Cycle Requirements 162
2.3.5.2 Rare Environmental Conditions 165
2.3.5.3 An Infinite Multiplicity of Combinations 166
2.3.6 Product-Specific Test Planning 166
2.3.6.1 Hardware 166
2.3.6.2 Application Specific Integrated Circuit (ASIC) 168
2.3.6.3 Firmware 170
2.3.6.4 Computer Software 170
2.3.6.4.1 Software Test Sequences 172
2.3.6.4.2 Software Test Methods 172
2.3.6.5 Personnel Procedures 175
2.3.7 Commercial Subset 175
Chapter 2.4 Item Qualification Test Implementation 178
2.4.1 Implementation Responsibility and Organizational Structure 178
2.4.2 General Coordination Activities 181
2.4.3 Individual Test Coordination 182
2.4.3.1 Authorization to Proceed 183
2.4.3.2 Test Preparation 184
2.4.3.3 Test Readiness Review 184
2.4.3.4 Test Task Go-Ahead 185
2.4.3.5 Test Conduct 186
2.4.3.6 Resolving Test Problems 186
2.4.3.7 Test Results Documentation and Review 188
2.4.3.8 Residual Disposition 189
2.4.4 Intertest and Intermethod Coordination 190
2.4.5 Qualification Test Sequencing 191
2.4.6 First Production Article Integration 194
Chapter 2.5 Nontest Item Qualification Methods Coordination 196
2.5.1 Other Methods and Meanings 196
2.5.2 Nontest Verification Responsibility 197
2.5.3 Nontest Documentation 197
2.5.4 Analysis 198
2.5.4.1 Specialty Engineering Requirements 198
2.5.4.1.1 System Safety 199
2.5.4.1.2 Reliability 200
2.5.4.1.3 Maintainability 201
2.5.4.1.4 Mass Properties 202
2.5.4.1.5 Specialty Engineering Verification Integration 202
2.5.4.2 Interface Requirements Verification Analysis 202
2.5.4.3 Environmental Requirements Verification Analysis 203
2.5.4.4 Analysis Strings 203
2.5.5 Demonstration 204
2.5.5.1 Specialty Engineering Requirements 204
2.5.5.2 Interface Requirements 204
2.5.5.3 Environmental Influences 204
2.5.5.4 Software Verification by Demonstration 205
2.5.6 Examination 205
2.5.6.1 Software Applications 205
2.5.6.2 Hardware Too 206
Chapter 2.6 Item Qualification Verification Reporting 208
2.6.1 Closing the Loop 208
2.6.2 Engineering Ethics 208
2.6.3 Report Structures 209
2.6.3.1 Integrated Verification Report Format 210
2.6.3.2 Independent Reports 210
2.6.3.3 Database Views 211
2.6.4 Report Content 211
2.6.5 Report Review, Approval, and Release 211
2.6.6 Report Capture and Availability 213
2.6.7 Report Source 213
2.6.7.1 Supplier Reporting and Access through the SDRL 213
2.6.7.2 Associate Reporting and Distribution 214
Chapter 2.7 Item Qualification Verification Management and Audit 216
2.7.1 Setting the Management Stage 216
2.7.2 The Management Agent 217
2.7.3 Documentation Overview 219
2.7.4 Task Integration and Scheduling 221
2.7.5 Task Readiness Review and Task Authorization 221
2.7.6 Evaluation and Refinement of Evidence 222
2.7.6.1 In-House Review of In-House Verification Reports 222
2.7.6.2 In-House Review of Supplier Verification Reports 222
2.7.6.3 Associate Interaction and Interface Verification Resolution 224
2.7.6.4 IV& V Witnessing and Review
2.7.6.5 Customer Coordination 225
2.7.7 Staging the FCA 227
2.7.7.1 Single or Multiple Audits 227
2.7.7.2 Audit Preparation 227
2.7.7.2.1 Completion Criteria 227
2.7.7.2.2 Audit Agenda 228
2.7.7.2.3 Agenda Responsibilities Assignment 228
2.7.7.2.4 Audit Venues and Resources 230
2.7.7.2.5 The Presentation Materials 230
2.7.7.2.6 Audit Dry Run 230
2.7.7.2.7 Audit Data Reproduction 231
2.7.8 Presentation of the Qualification Results: The Audit 231
2.7.9 Postaudit Action 232
2.7.10 Information Resources 232
Chapter 2.8 Item Qualification Closeout 236
2.8.1 System Verification Overview 236
2.8.2 System Verification Audit 238
2.8.2.1 System Audit Planning 238
2.8.2.2 The Audit 238
2.8.2.2.1 Item Verification Summaries and Integration 239
2.8.2.2.2 Item Audit Action Item Closeout Status 239
2.8.2.2.3 Interface Verification Audit 239
2.8.2.2.4 System Testing Results 240
2.8.3 Audit Follow-Up and Closeout 240
Part 3 ITEM ACCEPTANCE VERIFICATION 242
Chapter 3.1 Acceptance Test Planning Analysis 244
3.1.1 The Notion of Acceptance 244
3.1.2 Where Are the Requirements? 245
3.1.3 How Does Detail and Performance Specification Content Differ? 245
3.1.4 Conversion to Verification Requirements 245
3.1.5 Acceptance Test Planning, Procedures, and Results Data Collection 249
3.1.6 Associate Contractor Relationships 249
3.1.7 Manufacturing or Test and Evaluation Driven Acceptance Testing 250
3.1.8 Information Management 250
3.1.9 Coordination between Acceptance Testing and Special Test Equipment (STE) 251
3.1.10 Relationship between Technical Data and Acceptance 251
3.1.11 Postdelivery Testing Applications 253
3.1.12 Intercontinental Acceptance 253
Chapter 3.2 Nontest Item Acceptance Methods Coordination 254
3.2.1 Organizational Responsibilities 254
3.2.2 The Coordination Task 255
3.2.3 Acceptance Task Matrix 256
3.2.4 Examination Cases 256
3.2.4.1 Quality Acceptance Examinations 256
3.2.4.2 Engineering Participation in Acceptance Examination 258
3.2.4.3 Software Acceptance Examination 258
3.2.5 Demonstration Cases 259
3.2.5.1 Logistics Demonstrations 259
3.2.5.2 Flight Demonstration 260
3.2.6 Analysis Cases 260
Chapter 3.3 Product Acceptance Verification Reporting 262
3.3.1 Program Type 262
3.3.2 The Two Steps of Acceptance 262
3.3.3 Significant Differences 263
3.3.4 Reporting Structures 263
3.3.5 First Article Acceptance 264
3.3.6 Recurring Acceptance Verification 264
Chapter 3.4 Product Verification Management and Audit 266
3.4.1 The Second Stage of Verification 266
3.4.2 The Beginning of Acceptance Verification 267
3.4.3 The Basis of Acceptance 267
3.4.4 Acceptance Documentation 269
3.4.5 Management of the Work 269
3.4.6 FRACAS 272
3.4.6.1 Acceptance Ethics 272
3.4.6.2 FRACAS Implementation 272
3.4.7 Physical Configuration Audit (PCA) 274
3.4.7.1 PCA Planning and Preparation 274
3.4.7.2 PCA Implementation 275
3.4.7.2.1 The Physically Oriented Audit 275
3.4.7.2.2 The Paper-Dominated Audit 275
3.4.7.2.3 Combined Mode Audit 275
3.4.7.3 Post-PCA Activity 275
3.4.8 Software Acceptance 276
Part 4 SYSTEM TEST AND EVALUATION 278
Chapter 4.1 System Verification Planning 280
4.1.1 The Beginning 280
4.1.2 System-Level Testing and Analysis Implementation 281
4.1.2.1 Product Cases 281
4.1.2.2 System Test Categories 283
4.1.2.2.1 Development Test and Evaluation (DT& E)
4.1.2.2.2 Operational Test and Evaluation (OT& E)
4.1.2.2.3 Initial or Interim Operational Test and Evaluation (IOT& E)
4.1.2.2.4 Follow-On Operational Test and Evaluation (FOT& E)
4.1.3 Other Forms of System Testing 287
4.1.3.1 Quality and Reliability Monitoring 287
4.1.3.2 System Compatibility Test 287
4.1.3.3 Certification 290
4.1.4 The Grand Plan 291
4.1.5 The Limits of Intellectual Thought 292
4.1.6 The Details 293
Chapter 4.2 System Test and Evaluation Implementation and Management 294
4.2.1 System Test and Evaluation in the Beginning 294
4.2.2 Instrumentation Management 294
4.2.3 Reaching a State of DT& E Readiness
4.2.4 A Normal Evolution 297
4.2.5 Flexibility in the Face of Obstacles 299
4.2.6 DT& E Reporting
4.2.7 Logistics Support in DT& E
4.2.8 OT& E
4.2.9 Test Results Applications 302
4.2.10 System Test and Evaluation Closure 303
Chapter 4.3 Item and System Reverification 304
4.3.1 Reverification Exposed 304
4.3.2 Project Phase Subsets 305
4.3.2.1 Preproduction Development 305
4.3.2.2 Manufacturing 306
4.3.2.3 Product Use 307
4.3.3 Temporal Considerations 307
4.3.4 Process Redesign 309
Part 5 PROCESS VALIDATION AND VERIFICATION 312
Chapter 5.1 Process Validation 314
5.1.1 Is There a Difference? 314
5.1.2 Process Planning 315
5.1.2.1 Completeness 315
5.1.2.2 Accounting for Planning Risks 320
5.1.3 Program Process Design 322
5.1.4 Expanding Our Mental Boundaries 322
Chapter 5.2 Process Verification 324
5.2.1 Process Verification Timing 324
5.2.2 Program and Functional Metrics 324
5.2.3 Use of C/SCS in Process Verification 325
5.2.4 Progressive Planning Improvements 326
5.2.5 Organizational Possibilities 326
5.2.6 Enterprise Standards-Driven Process Verification 330
5.2.7 Total Quality Management 330
Part 6 POSTSCRIPT 332
Chapter 6.1 Verification and Systems Development Closure 334
6.1.1 Postscript Plan 334
6.1.2 Closure on Meanings 334
6.1.3 Hopes for Balanced Treatment 335
6.1.4 Information Opportunities 336
6.1.5 The Role of the System Engineer 337
6.1.6 Singling Up the WBS Lines 340
6.1.7 Focusing on the Whole 346
Chapter 6.2 Those Troublesome V Words 350
6.2.1 Introduction 350
6.2.2 The Contractual Difference 351
6.2.3 The JOG System Engineering V Word Set 352
6.2.3.1 Concept Validation 352
6.2.3.2 Requirements Validation 353
6.2.3.3 Verification 353
6.2.3.3.1 Item Qualification Verification 354
6.2.3.3.2 Item Acceptance Verification 354
6.2.3.3.3 System Test and Evaluation Verification 354
6.2.3.4 Technical Data 355
6.2.3.4.1 Technical Data Validation 355
6.2.3.4.2 Technical Data Verification 355
6.2.3.5 Product Representations 355
6.2.3.5.1 Representations Validation 355
6.2.3.5.2 Representations Verification 355
6.2.3.6 Certification 356
6.2.4 Toward a Generic Process 356
6.2.5 Final Closure on Meanings 361
6.2.6 Final Closure 362
INDEX 364