Survey Research and Questionnaires

Survey Research

Survey research is a commonly used method of collecting information about a population of interest. There are many different types of surveys, several ways to administer them, and many methods of sampling. There are two key features of survey research:

  • Questionnaires -- a predefined series of questions used to collect information from individuals
  • Sampling -- a technique in which a subgroup of the population is selected to answer the survey questions; the information collected can be generalized to the entire population of interest

Questionnaire Design

The two most common types of survey questions are closed-ended questions and open-ended questions.

Closed-Ended Questions

  • The respondents are given a list of predetermined responses from which to choose their answer
  • The list of responses should include every possible response and the meaning of the responses should not overlap
  • An example of a close-ended survey question would be, "Please rate how strongly you agree or disagree with the following statement: 'I feel good about my work on the job.' Do you strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, or strongly disagree?"
  • A Likert scale, which is used in the example above, is a commonly used set of responses for closed-ended questions
  • Closed-ended questions are usually preferred in survey research because of the ease of counting the frequency of each response

Open-Ended Questions

  • Survey respondents are asked to answer each question in their own words
  • Responses are usually categorized into a smaller list of responses that can be counted by the study team for statistical analysis

Considerations for Designing a Questionnaire

  • It is important to consider the order in which questions are presented. Sensitive questions, such as questions about income, drug use, or sexual activity, should be put at the end of the survey. This allows the researcher to establish trust before asking questions that might embarrass respondents. Researchers also recommend putting routine questions, such as age, gender, and marital status, at the end of the questionnaire
  • Double-barreled questions, which ask two questions in one, should never be used in a survey. An example of a double barreled question is, "Please rate how strongly you agree or disagree with the following statement: 'I feel good about my work on the job, and I get along well with others at work.'" This question is problematic because survey respondents are asked to give one response for two questions
  • Researchers should avoid using emotionally loaded or biased words and phrases

Visit the following websites for more information about questionnaire design:

Glossary terms related to questionnaire design:

Double-Barreled Question
Pretesting
Questionnaire

Survey Administration

Surveys can be admininistered in three ways:

  • Through the mail
    • Advantage: Low cost
    • Disadvantage: Low response rate
  • By telephone
    • Advantages: Higher response rates; responses can be gathered more quickly
    • Disadvantage: More expensive than mail surveys
  • Face-to-face
    • Advantages: Highest response rates; better suited to collecting complex information
    • Disadvantage: Very expensive

Visit the following website for more information about survey administration:

Glossary terms related to survey administration:

Attrition
Completion Rate
Cooperation Rate
Refusal Rate
Response Categories
Response Rate

Sampling Procedures

One of the primary strengths of sampling is that accurate estimates of a population's characteristics can be obtained by surveying a small proportion of the population. Four sampling techniques are described here:

Simple Random Sampling

  • Simple random sampling is the most basic form of sampling
  • Every member of the population has an equal chance of being selected
  • This sampling process is similar to a lottery: the entire population of interest could be selected for the survey, but only a few are chosen at random
  • Researchers often use random-digit dialing to perform simple random sampling. In this procedure, telephone numbers are generated by a computer at random and called to identify individuals to participate in the survey

Cluster Sampling

  • Cluster sampling is generally used when it is geographically impossible to undertake a simple random sample
  • Cluster sampling requires that adjustments be made in statistical analyses

For example, in a face-to-face interview, it is difficult and expensive to survey households across the nation. Instead, researchers will randomly select geographic areas (for example, counties), then randomly select households within these areas. This creates a cluster sample, in which respondents are clustered together geographically.

Stratified Sampling

  • Stratified samples are used when a researcher wants to ensure that there are enough respondents with certain characteristics in the sample
  • The researcher first identifies the people in the population who have the desired characteristics, then randomly selects a sample of them
  • Stratified sampling requires that adjustments be made in statistical analyses

For example, a researcher may want to compare survey responses of African-Americans and Caucasians. To ensure that there are enough Afrian-Americans in the survey, the researcher will first identify the African-Americans in the population and then randomly select a sample of African-Americans.

Nonrandom Sampling

  • Common nonrandom sampling techniques include convenience sampling and snowball sampling
  • Nonrandom samples cannot be generalized to the population of interest. Consequently, it is problematic to make inferences about the population
  • In survey research, random, cluster, or stratified samples are preferable

Visit the following websites for more information about sampling procedures:

Sampling

Glossary terms related to sampling procedures:

Convenience Sampling
Oversampling
Probability Sampling
Purposive sampling
Quota Sampling
Random Sampling
Random Selection
Representativeness
Sample
Sample Size
Sampling
Sampling Design
Sampling Frame
Snowball Sampling
Stratification
Stratified Sampling

Measurement Error

Measurement error is the difference between the target population's characteristics and the measurement of these characteristics in a survey. There are two types of measurement error: systematic error and random error.

Systematic Error

  • Systematic error is more serious than random error
  • Occurs when the survey responses are systematically different from the target population responses
  • For example, if a researcher only surveyed individuals who answered their phone between 9 and 5, Monday through Friday, the survey results would be biased toward individuals who are unemployed
  • Sources of bias include
    • Nonobservational error -- Individuals in the target population are systematically excluded from the sample, such as in the example above
    • Observational error -- When respondents systematically answer surveys question incorrectly. For example, surveys that ask respondents how much they weigh will probably underestimate the population's weight because respondents are likely to underreport their weight

Random Error

  • Random error is an expected part of survey research, and statistical techniques are designed to account for this sort of measurement error
  • Occurs because of natural and uncontrollable variations in the survey process, i.e., the mood of the respondent

For example, a researcher may administer a survey about marital happiness. However, some respondents may have had a fight with their spouse the evening prior to the survey, while other respondents' spouses may have cooked the respondent's favorite meal. The survey responses will be affected by the random day on which the respondents were chosen to participate in the study. With random error, the positive and negative influences on the survey measure balance out.

Visit the following website for more information about measurement error:

Glossary terms related to measurement error:

Interviewer Error
Nonsampling Error
Nonresponse Error
Nonresponse Rate Bias
Sampling Bias

Ethics of Survey Research

Informed Consent

Respondents should give informed consent before participating in a survey. In order for respondents to give informed consent,

  • The researcher must inform the respondents of the study's purpose, content, duration, and potential risks and benefits
  • The researcher must inform the respondents that they do not have to answer all the survey questions
  • The researcher must inform the resondents that they can stop participating in the study at any point

Confidentiality and Anonymity

It is absolutely imperative that researchers keep respondents' identities confidential. To ensure confidentiality, researchers should not link respondents' identifiers to their survey responses when using data. Common identifiers include names, social security numbers, addresses, and telephone numbers.

Anonymity

Anonymity is an even stronger safeguard of respondent privacy. If a researcher assures anonymity, it means that the researcher is unable to link respondents' names to their surveys.

Visit the following websites for more information about anonymity:

Glossary terms related to ethics:

Anonymity
Confidentiality
Informed Consent

Advantages and Disadvantages of Survey Research

Advantages

  • Sample surveys are a cost-effective and efficient means of gathering information about a population
  • Survey sampling makes it possible to accurately estimate the characteristics of a target population without interviewing all members of the population

Survey sampling is particularly useful when the population of interest is very large or dispersed across a large geographic area.

Disadvantages

  • Surveys do not allow researchers to develop an intimate understanding of individual circumstances or the local culture that may be the root cause of respondent behavior
  • Respondents often will not share sensitive information in the survey format
  • A growing problem in survey research is the widespread decline in response rates

Research Connections is supported by grant #90YE0104 from the Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. The contents are solely the responsibility of the National Center for Children in Poverty and the Inter-university Consortium for Political and Social Research and do not necessarily represent the official views of the Office of Planning, Research and Evaluation, the Administration for Children and Families, or the U.S. Department of Health and Human Services.

Google Translate