Intervention Effectiveness Research: Quality Improvement and Program Evaluation (eBook)

Karen A. Monsen, PhD, RN, FAAN, is an award-winning scholar and expert nurse informatician.  She is co-director of the University of Minnesota Center for Nursing Informatics, Director of the Omaha System Partnership, affiliate faculty in the Institute for Health Informatics, and faculty in the Center for Spirituality and Healing. Dr. Monsen specializes in data management for intervention effectiveness research, quality improvement, and outcome evaluation in nursing and health care.  Based on her years of public health experience and her insights from exploring big data methods with existing datasets, Dr. Monsen celebrates the power of shared understanding enabled by the use of standardized data. At the cross roads of research and practice, she champions intervention effectiveness research through three major initiatives: The Omaha System Community of Practice, The Omaha System Partnership for Knowledge Discovery and Health Care Quality; and The Omaha System Guidelines. These initiatives are face-to-face and Internet forums for research and quality improvement projects world-wide.  Dr. Monsen is the author of over 70 published studies that have advanced the design, development and evaluation of clinical practice for diverse problems, programs, and populations.

Foreword 6
Preface 8
Abbreviations 17
Contents 10
Part I: Introduction to Intervention Effectiveness Research, Quality Improvement, and Program Evaluation 18
1: Key Concepts, Definitions, and Frameworks 19
1.1 Introduction 19
1.2 Definitions and Descriptions of Intervention Effectiveness Research, Quality Improvement, and Program Evaluation: What They Have in Common and How They Differ 20
1.2.1 What Is Quality Improvement? 20
1.2.2 What Is Program Evaluation? 21
1.3 How Intervention Effectiveness Research, Quality Improvement Activities, and Program Evaluation Are Similar 21
1.4 How Intervention Effectiveness Research, Quality Improvement, and Outcome Evaluation Are Different 22
1.4.1 Translational Research 22
1.4.2 Quality Improvement (QI) 22
1.4.3 Six Sigma Quality Improvement 23
1.4.4 Health Services Research 23
1.4.5 Big Data in Health Care Research 24
1.4.6 Program Evaluation 24
1.4.7 Implementation Research 25
1.5 Definitions of Similar Sounding Terms and What This Book Does Not Attempt 25
1.5.1 Comparative Effectiveness Research 25
1.5.2 Implementation Science Research 25
1.5.3 Dissemination Science 26
1.6 Frameworks to Support Intervention Effectiveness Research, Quality Improvement Activities, and Program Evaluation 26
1.6.1 Theory 26
1.6.2 Logic Models 28
1.6.3 Theoretical Framework 28
1.6.4 Conceptual Framework 28
References 29
2: Problem-Intervention-Outcome Meta-­Model (PIO MM): A Conceptual Meta Model for Intervention Effectiveness Research, Quality Improvement Activities, and Program Evaluation 32
2.1 Introduction to the Problem-Intervention-Outcome Meta-Model (PIO MM) 32
2.2 PIO MM and the CDC Logic Model 34
2.3 PIO MM and the IHI Quality Improvement Model 35
2.4 Using the PIO MM 37
2.5 Operationalizing the PIO MM 40
2.6 PIO MM Relationship to Change Theory 41
2.7 PIO MM Relationship to PICOT 41
References 42
3: Problem-Intervention-Outcome Meta-Model Project Design 44
3.1 Design for Intervention Effectiveness Research, Quality Improvement Activities, and Program Evaluation 44
3.1.1 Observational Design 44
3.1.2 Retrospective Design 45
3.1.3 Prospective Design 46
3.2 Intervention and Measurement Timing 47
3.3 PIO MM and Research Design 47
3.4 Benefits and Challenges of the Single Group Before and After Design 47
3.4.1 Threats to Internal Validity 48
3.4.2 Enhancing Before and After Design Using Comparisons 48
3.4.3 Considerations for Prospective Data Collection 49
3.5 Comparisons Using PIO MM Variables 49
3.5.1 Problem 49
3.5.2 Intervention 49
3.5.3 Interventionist 50
3.5.4 Outcome 51
3.5.5 Population (Individual Characteristics) 51
3.5.6 Setting 51
3.5.7 Time 52
3.6 Mixed Methods: Qualitative Evaluation 52
References 53
4: Tools for Intervention Effectiveness Research, Quality Improvement Activities, and Program Evaluation 55
4.1 Data Sources 55
4.2 Checklists for Obtaining New or Existing Data for Operationalizing the PIO MM 56
4.3 Electronic Health Record Data 58
4.4 Nursing-Specific Data 58
4.5 Omaha System 59
4.5.1 Problem Classification Scheme 60
4.5.2 Intervention Scheme 60
4.5.3 Problem Rating Scale for Outcomes 60
4.6 Analysis Software and Techniques 62
4.7 Power Analysis 62
4.8 Software for Descriptive and Inferential Statistical Methods and for Creating Graphs/Charts 62
4.8.1 Microsoft Excel 62
4.8.2 R 63
4.8.3 SAS 63
4.9 Big Data (Pattern Detection) Methods 63
4.9.1 Clustering 64
4.9.2 Visualization 64
4.10 Team Approach 65
References 65
5: Descriptive Analysis and Interpretation 67
5.1 Introduction 67
5.2 Data Cleaning 67
5.2.1 Screening Phase 68
5.2.2 Diagnostic Phase 68
5.2.3 Treatment Phase 69
5.2.4 Missing Data 69
5.3 Pre-Processing 69
5.3.1 Transforming and Recoding 69
5.3.2 Identification and Labeling of Clusters Within a Sample 70
5.4 Descriptive Statistics 71
5.4.1 Frequency 72
5.4.2 Cross Tabulation (Cross Tab) Matrix 73
5.4.3 Rank 73
5.4.4 Measures of Central Tendency 73
5.4.5 Measures of Distribution 75
References 76
6: Inferential Analysis and Interpretation 77
6.1 About Inferential Statistics 77
6.2 Comparisons and Statistical Significance 80
6.2.1 Comparisons of Sample Characteristics 80
6.2.2 Outcomes as Measured by Before and After Comparison 80
6.2.3 Benchmarking 82
6.2.4 The P-Value in Large Dataset Research 83
6.3 Clinical or Practical Significance 83
6.3.1 Effect Size (Clinical or Practical Significance of Pchange = PTime2 ? PTime1) 83
6.3.2 Interpretation of Effect Size (Clinical or Practical Significance) 84
6.4 Associations 84
6.4.1 Correlation 84
6.4.2 Regression 86
6.4.3 Interpretation of Correlations 86
6.4.4 Survival Analysis (PTime1, PTime2, … PTimeX) 86
6.4.5 Cross Tabs and Chi-Square (?2) 87
6.5 Generalizability 87
References 89
7: Exploratory Data Analysis 90
7.1 The Development of Exploratory Data Analysis 90
7.2 Interpretation of Exploratory Data Analysis 91
7.3 Visualization Techniques 91
7.3.1 Heat Map 91
7.3.2 Line Graph 93
Line Graph with Trend Line 95
Parallel Coordinates 95
References 97
8: Ethical Considerations 99
8.1 Minimal Risk 99
8.2 Institutional Review 100
8.2.1 Where and How to Access an IRB 100
8.2.2 When a Project May Be Exempt from IRB Review 100
8.2.3 The Special Case of Quality Improvement 101
8.2.4 Minimal Risk and IRB Review 102
8.2.5 The Special Case of Program Evaluation 103
8.3 Informed Consent 104
8.3.1 What Is Informed Consent? 104
8.3.2 Informed Consent Processes in the Context of Existing Data 105
8.4 Data Privacy and Security 105
References 107
Part II: Practical Guide for Using the Problem-­Intervention-­Outcome Meta-Model 108
9: Use the Worksheets and PIO MM Figure 109
9.1 Review of Part I 109
9.2 Overview of Part II 110
9.2.1 Examples of Projects 110
9.3 Starting the Process 111
9.3.1 Worksheet Review 111
9.3.2 Complete the PIO MM Diagram 115
References 115
10: Know the Literature (Worksheet A) 116
10.1 Preparing to Complete Worksheet A 116
10.2 Step-by Step Instructions for Completing Worksheet A 117
10.2.1 Population of Interest 118
10.2.2 Problem Addressed 119
10.2.3 Measure(s) of Outcome 119
10.2.4 Intervention(s) Used 120
10.2.5 Measures of Intervention 121
10.2.6 Measure of Intervention Fidelity 121
10.2.7 Demographic Characteristics of a Sample 122
10.2.8 Contextual Factors 122
10.2.8.1 Contextual Factors – Interventionist 123
10.2.9 Analysis Methods 123
10.2.10 Comments 124
10.2.11 Complete Reference 124
10.3 Sources of Information for the PIO MM Matrix 124
References 127
11: Define the Problem (Worksheet B) 128
11.1 Preparing to Complete Worksheet B 128
11.2 Step-by Step Instructions for Completing Worksheet B 129
11.2.1 Problem 129
11.2.2 Definition of the Problem 130
11.2.3 Population of Interest 130
11.2.4 Background 130
11.2.5 Problem Measurement Instrument/Scale 131
11.2.6 Anticipated Outcome and Rationale 132
11.2.7 What is Not Known/Gap in Knowledge 132
References 137
12: Describe the Intervention (Worksheet C) 139
12.1 Preparing to Complete Worksheet C 139
12.2 Step-by Step Instructions for Completing Worksheet C 140
12.2.1 Describe the Intervention 140
12.2.2 Expected Effectiveness 140
12.2.3 Theoretical Framework 141
12.2.4 Intervention Content and Essential Core Components 141
12.2.5 Describe Intervention Measurement: Amount, Type, Fidelity, Quality 142
12.2.6 Describe Interventionist Characteristics: Qualifications, Training, Demographics 143
References 149
13: Define the Outcome (Worksheet D) 151
13.1 Preparing to Complete Worksheet D 151
13.2 Step-by-Step Instructions for Completing Worksheet D 152
References 160
14: Plan the Analysis (Worksheet E) 162
14.1 Preparing to Complete Worksheet E 162
14.1.1 Step 1. Review Project Statements 163
14.1.2 Step 2. Select Statements That Are Most Applicable to the Project and Discipline 165
14.1.3 Step 3. Review Design Options 165
14.1.4 Step 4. State the Design 165
14.1.5 Step 5. Review Variables 166
14.1.6 Step 6. Plan for Creating New Variables 166
14.2 Step-by-Step Instructions for Completing Worksheet E 167
14.2.1 Exploratory Data Analysis 167
14.2.2 Sample 167
14.2.3 Intervention 168
14.2.4 Outcome 169
14.2.5 Relationships Among Variables 170
References 171
15: Interpret the Results (Worksheet F) 173
15.1 Preparing to Complete Worksheet F 173
15.2 Results Statements and Presentation 174
15.2.1 Presenting the Results 174
15.2.2 Description of Sample Characteristics 174
15.2.3 Description of Interventions 176
15.2.4 Description of Outcomes 177
15.2.5 Description of Benchmark Attainment 179
15.2.6 Correlations Between Interventions and Outcomes 180
15.3 Results Interpretation 181
15.3.1 Theoretical Framework-Related Interpretation 182
15.3.2 Temporality-Related Interpretation 182
15.3.3 Give Alternative Explanations for the Findings 183
References 184
16: Disseminate the Findings 185
16.1 Why Dissemination Matters 185
16.2 Getting the Most Benefit from This Chapter 186
16.3 Iterative Interpretation and Explication of the Overall Story 186
16.4 Drafting the Abstract: Summarize the Story in Brief 190
16.5 Develop and Display Results 191
16.6 Adding Meaningful Interpretation to the Results 193
16.7 Limitations 199
16.8 The Methods Section 200
16.9 The Purpose Statement 201
16.10 Background to Set the Stage for the Purpose 201
16.11 The Gap in Knowledge 203
16.12 Title, Abstract, and Conclusion 204
16.13 Rewrite the Abstract 205
16.14 Write the Conclusions Section 205
16.15 Polishing Tips 207
16.16 Styles and Author Guidelines 207
References 208
17: Synthesis, Next Steps, and Epilogue 210
17.1 Planning Next Steps 210
17.2 Questions to Inspire Next Steps 212
17.3 Building Evidence on Evidence 213
References 214