Observational Measurement of Behavior

Dr Paul J. Yoder, Ph.D. Professor, Department of Special Education, Peabody College, Vanderbilt University, Nashville, Tennessee 37203 Dr. Paul Yoder has been studying the transition from prelinguistic to linguistic communication in multiple populations with disabilities for over two decades. He is a co-designer of Milieu Communication Teaching and has contributed to several studies examining the efficacy of this treatment. He teaches methods and measurement at Vanderbilt University. Primary research activities of Frank J. Symons, Ph.D., are supported by the National Institute of Child Health and Human Development (NICHD), and they focus on improving the assessment and treatment of severe self-injurious behavior among individuals with developmental disabilities and pervasive developmental disorders. Dr. Symons was a research scientist at the Frank Porter Graham Child Development Center at the University of North Carolina at Chapel Hill and a postdoctoral fellow at the John F. Kennedy Center at the Peabody College of Vanderbilt University in Nashville, Tennessee. He is the co-author of Behavioral Observation: Technology and Applications in Developmental Disabilities (Paul H. Brookes Publishing Co., 2000).

1: Introduction and Measurement Contexts
Overview
Definition of Systematic Observation Using Count Coding
Rationale for Systematic Observation Using Count Coding
Importance of Falsifiable Hypotheses
The Continuum of State-Likeness to Trait-Likeness
Context-Dependent Behavior
Person Characteristics
Generalized Behavioral Tendencies
Skills
The Relative Scientific Value of Different Objects of Measurement
Ecological Validity and Representativeness
Conclusions and Recommendations
References
2: Validation of Observational Variables
Overview
The Changing Concept of Validation
Understanding Which Types of Validation Evidence Are Most Relevant for Different Research Designs, Objects of Measurement, and Research Purposes
Content Validation
Definition of Content Validation
Different Traditions Vary on the Levels of Importance Placed on Content Validation
Weaknesses of Content Validation
Sensitivity to Change
Definition of Sensitivity to Change
Influences on Sensitivity to Change
Weaknesses of Sensitivity to Change
Treatment Utility
Definition of Treatment Utility
Weaknesses of Treatment Utility
Criterion-Related Validation
Definition of Criterion-Related Validation
Primary Appeal of Criterion-Related Validation
Weaknesses of Criterion-Related Validation
Construct Validation
Definition of Construct Validation
Discriminative Validation
Nomological Validation
Multitrait, Multimethod Validation
An Implicit “Weakness” of Science?
Recommendations
References
3: Measuring Person Characteristics
Overview
Contextual Measurement Error
Definition of Measurement Context
A Brief Overview of Measurement Theory
Definition of Contextual Measurement Error
Representativeness
Contextual Measurement Error in Measures Of Generalized Behavioral Tendencies
Averaging Scores Across Contexts Improves Measures of Generalized Behavioral Tendencies
Aggregates Tend to Improve Estimates of Known True Score.
Aggregates Tend to Improve Construct Validity.
Aggregates Tend to Improve Stability.
Controlling Influential Contextual Variables Stabilizes Observed Scores for Highly Variable Person Characteristics
Why Naturalistic Observations Are Not Necessarily More Representative Than Contrived Ones
Why Skills Are Often Measured in Structured Measurement Contexts
Why Skills Are Often Assessed in Clinics or Labs
The Link Between Stability and Construct Validity
Recommendations and Conclusions
References
4: Designing or Adapting Coding Manuals
Overview
Definition of a Coding Manual
Deciding Whether to Use an Existing Coding Manual or to Construct a New One
Recommended Steps for Modifying or Designing Coding Manuals
Defining When to Start and Stop Coding
Conceptually Defining the Context-Dependent Behavior or the Generalized Characteristic
Defining the Highest Level of Codable Behavior
Determining the Level of Distinction Coders Have to Make
Organizing the Coded Categories into Mutually-Exclusive Sets
Physically Based Definitions, Socially Based Definitions, or Both?
Defining the Lowest Level Categories
Source of Conceptual and Operational Definitions
A Qualitative Approach to Identifying Definitions
Defining Segmenting Rules
The Potential Value of Flowcharts
Do Coding Manuals Need to be Sufficiently Short to be Included in Methods Sections?
Recommendations and Conclusions
References
5: Coding
Overview
The Elements of an Observational Measurement System
Behavior Sampling
The Superordinate Distinctions: Continuous versus Intermittent
The Subordinate Distinctions: Timed-Event versus Event versus Interval
Timed-Event Sampling
Event Sampling
Interval Sampling
Types of Interval Sampling
Whole Interval Sampling
Momentary Interval Sampling
Partial Interval Sampling
The Importance of Knowing What Metric the Investigator Wants to Estimate
Summary of Behavior Sampling
Participant Sampling
Focal Sampling
Multiple Pass Sampling
Conspicuous Sampling
Reactivity
Live Coding versus Recording the Observation For Later Coding
Live Coding
Coding from Recorded Sessions
Recording Coding Decisions
Recommendations and Conclusions
References
6: Common Metrics of Observational Variables
Overview
Definition of Metric
Quantifiable Dimensions of Behavior
Proportion Metrics
Proportion Metrics Change the Meaning of Observational Variables
Scrutinizing Proportions
An Implicit Assumption of Proportion Metrics
Testing Whether the Data Fit the Assumption of Proportion Metrics
Consequences of Using a Proportion When the Data Do Not Fit the Assumption
Alternative Methods to Control Influential Contextual Variables
Statistical Control
Procedural Control
Aggregate Measures of Generalized Person Characteristics
Weighted Count
Unit-Weighted Aggregates
Group Analysis of Observational Variables
Transforming the Metric
Bootstrapping
Recommendations and Conclusions
References
7: Observer Training and Preventing Observer Drift
Overview
Point-by-point Agreement and Disagreement
Point-by-point Agreement of Interval Sampled Data
Point-by-point Agreement of Timed-Event Data
Discrepancy Matrices
Discrepancy Discussions
Using Discrepancy Discussions to Train Observers
Creating Criterion-coding Standards.
Remaining Steps to Train Observers
Preventing Observer Drift
Method of Selecting Sessions for Agreement Checks
Remaining Steps to Preventing or Addressing Observer Drift
Recommendations
References
8: Interobserver Agreement and Reliability of Observational Variables
Overview
Additional Purposes of Point-by-Point Agreement
Added Principles When Agreement Checks Are Used to Estimate Interobserver “Reliability” of Observational Variable Scores
Exhaustive Coding Spaces Revisited
The Effect of Chance on Agreement
Common Indices of Point-by-Point Agreement
Occurrence Percentage Agreement
Nonoccurrence Percentage Agreement
Total Percentage Agreement
Kappa
Base Rate and Chance Agreement Revisited
Intraclass Correlation Coefficient (ICC) as an Index of Interobserver Reliability in Group Designs
Options for Running ICC With SPSS
Between-Participant Variance on the Variable of Interest Affects ICC
Using ICC as a Measure of Interobserver Reliability for Predictors and Dependent Variables in Group Designs
The Interpretation of SPSS Output for ICC
The Conceptual Relation Between Interobserver Agreement and ICC
Consequences of Low or Unknown Interobserver Reliability
Recommendations
References
9: Introduction to Sequential Analysis
Overview
Definition of Terms Used in this Chapter
Sequential versus Nonsequential Variables
Sequential Associations are not Sufficient Evidence for Causal Inferences
Coded Units and Exhaustiveness
Contingency Tables
Three Major Types of Sequential Analysis
Event Lag
Concurrent Interval
Event Lag with Pauses (to replace time window method)
Explanation for no longer focusing on time window
Indices of Sequential Association: Controlling for Chance
Existing Indices of Sequential Association: Advantages and Disadvantages
Transitional/conditional probabilities
Yule’s Q
Risk Difference/Operant Contingency Value
Other commonly-used indices and why we do not focus on them (e.g., z)
Recommendations and Conclusions
References
10: Identifying and Addressing Research Questions Involving Sequential Associations
Overview
Sequential Analysis in Group Designs
Types of Research Questions and Methods to Address Them
Indices of Sequential Association as Dependent Variables
Testing the Significance of a Mean Sequential Association
Testing the Between-Group Difference in Mean Sequential Associations
Testing the Within-Subject Difference in Sequential Associations
Sequential Analysis in Single-Case Designs
Types of Research Questions
Descriptive Questions Involving Behavior-Environment Associations to Inform Experimental Analyses
Descriptive Questions to Inform Temporal Distribution of One or More Target Behaviors
Indices of Sequential Association as Dependent Variables in Single-Case Designs
Indices of Sequential Association to Inform Procedural Fidelity
Methods to Address Sequential Analysis Research Questions in a Single Case Framework
Why Significance Testing is Inappropriate at the Level of the Individual Participant
More on the Term `Operant Contingency’
Summary of Analysis Methods Used in Behavior Analytic Literature
Conditional Probabilities
Lag Sequential Analysis
Contingency Space Analysis
What is “Enough Data” and How Do We Attain It?
Proposed Solutions for Insufficient Data
Summary and Recommendations
References
11: Generalizability Theory
Overview
Scope of This Chapter
Overview of G Theory and Definition of Terms
An Example Observer by Context G and D Study
Rationale for Preferring the Absolute G Coefficient
Example Applications of D Studies
An Ongoing Controversy
Recommendations and Conclusions
References
12: Best Practices in Observational Measurement
Glossary
Index