UNIX Test Tools and Benchmarks

Preface. TESTING METHODS AND DEFINITIONS. 1. Functional, Structural, Reliability, and Regression Testing Overview. Motivation for Writing this Book. Functional, Structural, Reliability and Regression Testing. Reasons for Testing. Challenges to Testing. The Purpose of Testing. Quality and Testing. Verification and Validation. Errors, Faults and Failures. A Testing Hierarchy and Test Assertions. Common Testing Problems. Automation, Planning and Stages of Testing. Test Plans (Step 1). Unit Testing (Step 2). Interfaces and Integration Testing (Step 3). System and Acceptance Testing (Step 4). Conclusions. 2. Functional Testing. Introduction to Functional Testing. Functional Test Coverage. The Structured Review Process (Functional Testing). Call Graphs and Top-Down Functional Testing. Functional Testing for Defect Containment. Standards Conformance and Functional Testing. Dirty and Clean Functional Tests. Functional Testing (Domains vs. States). Domain Testing and GUI Environments. Automated Test Generation. Orthogonal Arrays for Functional Test Selection. Call Graphs and Touch Tests. Monkey, Random and Touch Testing. Conclusions. 3. Structural Testing and Structural Analysis. Introduction to Structural Testing and Analysis. Structural Analysis. Static Analysis. Metrics and Testing. A Rifle Shot Approach. Dynamic and Static Structural Analysis. Memory Management Tools. Be Careful With The Data! Conclusions. 4. Reliability Testing. Introduction to Reliability Testing. Testability and Reliability. Testability of Object-Oriented Software. Run Reliability. Reliability and CM. Stress and Load Testing For Reliability. Exercisers. Tips for Stress and Load Tests. Test Harnesses and Load/Stress Testing. Multiprocessor and Multi-tasking Considerations for Reliability Testing. How Much Time Does Load Testing Require? Benchmarking Reliability. Mean Time Between Failure (MTBF) and Software Reliability. Conclusions. 5. Regression Testing. Introduction to Regression Testing. Regression Testing Approaches. Regression Testing and QA. Video-taping and Regression Testing. Regression Test Reports. The Baseline Build Process (BBP) and Regression Testing. A Fully Automated System. Computer-Aided Software Test (Regression Testing). Systematic Testing (Context Free Capture/Playback). Defect Tracking Systems, CM, and Regression Testing. Conclusions. 6. Metrics. Introduction to Metrics. When Defects are Discovered. Caveats to Metrics. Bug Categories. Defect Density and Defect Containment. Test Coverage. Bad Fixes (Regressions). Change History. Size, Cost and Effort. Documentation Problems. Usability. Mean Time Between Failure (MTBF). Call Entry and Exit Coverage with Interfaces. Project Completion Time. No Single Metric. Garbage In Garbage Out (GIGO). Who is Involved? Other Metrics for Measurement. Conclusions. 7. Load and Stress Testing. Introduction to Load and Stress Testing. Stress Testing Definitions. Difficulties with Stress Testing. UNIX Commands for Measuring System Stress and Load. Time and Resource Requirements of Stress Tests. Modular Tests. Tape and Floppy Exercisers. Load Testing. Ensembles of Tests. Fighting the Clock. False Alarms. The Kitchen Sink Approach - NOT! Source Code Control and CM. Performance Metrics and Load Testing. Exercisers vs. Functional Tests. Software Resource Exhaustion Tests. System Exercisers. User Exercisers. Final Stress and Load Test Tips. Conclusions. 8. Quality Assurance. Introduction to QA. The Need For Quality Data. Feature Content, Schedule, People, Equipment and Quality. Software QA and Quality Control Quality Group Responsibilities Benefits Associated with Software QA Difficulties Associated with Software QA Conclusions TEST TECHNOLOGY. 9. Standards Conformance Test Suites. Introduction to Standards Conformance Test Suites. Industry Standards, QA and Strategic Planning. UNIX Unification. API and ABI Standards Testing. ABI Standards Testing (A Reference Platform). Testing and Standards Conformance Conflicts. Hardware and Software Certification. Branding and Certification. System V Verification Suite (SVVS4). Application Binary Interface (ABI) Compatibility. SVVS4 (Dependency Issues). Verification Suite X/Open (VSX). XPG Branding. X/Open Test Suites. Setup and Administration of VSX. generic Application Binary Interface (gABI). ApTest. Sparc International (SI). POSIX Conformance Test Suite (PCTS). Conclusions. 10. Validation Test Suites. Introduction Validation Test Suites. Decision Coverage and Validation Test Suites. The Need for Validation Test Suites. UNIX Validation Suite 5.3 (UVS3). UNIX Validation Suite 5.4 (UVS4). UNIX Validation Suite BSD (UVSE). ARTUS Buster. X Window System Test Suite (xtest). NFS Test Suite (connectathon). Conclusions. 11. Language Validation Test Suites. Introduction to Language Validation Test Suites. The ANSI/ISO C Validation Suite (ACVS). The C Validation Suite (CVSA). The C++ Validation Suite (C++VS). FORTRAN Conformance Validation Suite (FCVS). ADA Validation Suites. Pascal Validation Suite (PVS). Plum Hall Validation Test Suites. Validation Suite For C. RoadTest C and C++ Compiler Test Suites. Conclusions. 12. Stress Tests. Introduction to Stress Tests. X Window Server Stress Test (xcrash). Stress Tests (Tape/Floppy). Disk Stress Test (disk). paranoia. UNIX File System Stress Suites. Conclusions. 13. Benchmarks. Introduction to Benchmarking. Performance Engineering. AIM Technology. AIM Suite III. AIM Suite V. AIM Milestone. LADDIS - NFS Benchmark (SPEC). SPEC. SPEC (SDM). SPEC92. bonnie. byte. dhrystone. disktest (Disk Input and Output Testing). iobench. iostone. iozone. linpack. musbus. nhfsstone. slalom. Stanford Integer Suite Benchmark. ttcp (now tsock). whetstone. WPI Benchmark. x11perf. xbench. Conclusions. 14. Test Tools. Introduction to Test Tools. Other Testing and Tool Resources. Assertion Definition Language (ADL). ATAC (Data and Control-Based Test Coverage for C Programs). Cantata. CAPBAK/NI and CAPBAK/X. CodePlan. DDTs (Distributed Defect Tracking system). Data Generation Language (DGL). Evaluator. EXDIFF and Spiff. expect TCL. FERRET (Automated Software Testing System). FlightRecorder. Generic Coverage Tool (gct). GNU Bug-Tracking System (GNATS). Insight++. Interactive Development Environments (IDE) - StP/T. IPS. METRIC. news, notes and MOSAIC. ParaSET. preVue-X. PiSCES. PureCoverage. Purify. QA Partner. QASE. QualityTEAM. Quantify. Remedy Corporation's Action Request (AR) System. ReplayXt. SMARTS. SoftTest. S-TCAT. TCAT. TCAT-PATH. tcov and Hindsight. Test Generator Generator System (TGGS). WinRunner and XRunner. ViSTA. X11 Virtual Display System. xtrap. Conclusions. 15. Test Harnesses. Introduction to Test Harnesses. Purpose. Available Test Harnesses. Testsuite Environment Toolkit (TET). TCC Operations. Perennial's"driver. DejaGnu. Internal Tools (Test Harnesses). Test Harnesses and POSIX. MCtst. TestExpert. TestDirector. Test Harness Architecture and Features. Benefits Of A Test Harness. Challenges and Problems (Verification). Methodology, Responsibilities, and Return on Investment. Conclusions. 16. Final Remarks. Test Architecture. Quality Assessment Survey. APPENDIX A. Floppy Exerciser. APPENDIX B. New Features for VSX4. APPENDIX C. FCVS Run Script. APPENDIX D. Alphabetical List of Companies. APPENDIX E. Verification and Validation Plan Outline and Checklist. Introduction and Purpose. Reference Documents and Test Items:. Testing Activities and Approach (Tables Are Recommended):. Customer Acceptance Tests. Standards, Practices, Conventions and Metrics. Problem reporting and corrective action:. Code Control (CM). Media Control. Supplier Control. Responsibilities. Staffing and Training. Schedule. Risks and Contingencies. Approvals. Verification & Validation Checklist. Testing Activities and Approach (Tables Recommended). Item Pass/Fail Criteria (Expected Results). Suspension and Resumption Criteria (Crashes, Hangs, etc.). Test Deliverables (Test Plans, Reports, Tools, Tests, etc.). Testing Tasks (Skills and Dependencies). Environmental Needs (Hardware, Software, Tools, Network, etc.). Customer Acceptance Tests. Standards, Practices and Metrics (Values are for Example Only). Approach (Techniques, Tools, Methods, Completion Criteria - Defect Density). Pre-release Quality Metrics. System Testing. Acceptance Test Procedure. APPENDIX F. Beta Test and Customer Acceptance Test Checklist. Introduction. Objectives. Ownership. Three Phases of Beta. Beta Test Best Practice Checklist. References. Glossary. Index.