Analysis of the Cognitive Interview in Questionnaire Design

It is common to conduct cognitive testing to evaluate survey questionnaires, but analysis procedures have not been well described. Dr. Willis describes alternative models of coding, analysis, and reporting, to enable researchers to convert cognitive-interview data into meaningful, well-supported, and actionable findings and summary reports.

---

Cognitive interviewing, based on the self-report methods of Ericsson and Simon, is a key form of qualitative research that has developed over the past thirty years. The primary objective of cognitive interviewing, also known as cognitive testing, is to understand the cognitive mechanisms underlying the survey-response process. An equally important aim is contributing to the development of best practices for writing survey questions that are well understood and that produce low levels of response error. In particular, an important applied objective is the evaluation of a particular set of questions, items, or other materials under development by questionnaire designers, to determine means for rewording, reordering, or reconceptualizing. Hence, as well as providing an empirical, psychologically oriented framework for the general study of questionnaire design, cognitive interviewing has been adopted as a 'production' mechanism for the improvement of a wide variety of survey questions, whether factual, behavioral, or attitudinal in nature.

As with other methods that rely on qualitative data, cognitive interviewing has increasingly been criticized for being lax in the critical area of the development of systematic methods for data reduction, analysis, and reporting of results. Practitioners tend to conduct cognitive interviewing in varying ways, and the data coding and compilation activities undertaken are often nonstandardized and poorly described. There is a considerable need for further development--and documentation--relating not only to a description of this variation but also to providing a set of recommendations for minimal standards, if not best practices. The proposed volume endeavors to address this clear omission.