• Devising a sampling plan: Who are the respondents and how many respondents do you need to provide reliable and valid results? If you have a bad sample, your data might lead the product team astray.

• Developing a data analysis plan: What types of data will help you meet your goals? What analyses will help you understand the raw data?

• Recruiting respondents: Recruiting the target respondents is often a struggle, especially with privacy, opt-out policies, anti-spam policies, and difficulties getting directly at users (as opposed to customers).

• Writing cover letters and introductions that explain your study: Good cover letters can influence response rates.

• Sending out follow-up reminders: Reminders can be quite effective at encouraging responses, but they need to be written to be both gentle and persuasive so the respondent is not irritated.

• Analyzing the data: For large-scale studies, it is useful to consult with a business analyst or statistician who can guide you through data analysis issues such as inflated alpha levels (when you do multiple statistical tests on the same set of data, you are more likely to find spurious significant results) and confidence intervals (a way to assess the uncertainty due to sampling around statistics, such as means).

• Interpreting the data: What do your data mean? Interpretation involves both art and science. You need to understand how the sampling and context of your study can affect your results.

• Presenting the data to stakeholders: What is the best approach for presenting your data? How much do you present? How much detail do you need to ensure credibility?

The Long History of Surveys and Questionnaires

One of the earliest surveys on record is described in the Domesday Book, which reported on a land survey commissioned in 1085 by William the Conqueror to determine who owned land and other resources in England. The survey results were used to determine how much tax could be extracted from the landowners.

Around 800 years later, in the 1870s, Sir Francis Galton, the Victorian polymath and cousin of Charles Darwin, introduced the modern questionnaire method into psychology with his controversial book, English Men of Science: Their Nature and Nurture (Galton, 1874). Galton used the questionnaire method to examine whether interest in science was due to heredity or environment by sending questionnaires to 190 members of the Royal Society and asking questions about environment, birth order, and characteristics of the parents of Society members.

• The questionnaire is tapping the concepts of interest.

• The potential respondents are competent sources of the information you need.

• The respondents are willing to commit the time to answer your questionnaire (this gets into relevance and extrinsic and intrinsic motivation).

• All of the respondents understand your questions in the same way.

• People will follow the rules of the survey and provide (to the best of their abilities) honest data.

• The structure of questions: When should you use open, closed, or mixed questions? Mixed questions include both specific responses and an open-ended “Other” response that allows respondents to write in an answer not provided.

• The wording of questions and responses: Changing a single word, even one that seems innocuous, could change the pattern of responses.

• Question order: Each question you ask can influence the answers to subsequent questions. In the extreme, asking something that is too threatening or personal at the beginning of a questionnaire might result in the respondent rejecting your survey immediately. Deciding how to order questions is, like other aspects of questionnaire design, part art and part science.

• Biases: Bias can be subtle or blatant and show up in the design of questionnaire layouts, instructions, questions, and responses. Here are a few examples of biases that can creep in questionnaire and survey design. Others will be discussed throughout this chapter.

• Primacy/recency: The tendency...