The Research Connections website is undergoing renovations. All data and resources will continue to be available while we work on the website. Sign up for the Research Connections newsletter to receive the latest updates. You can also watch our video.

Data Collection

Survey Research and Questionnaires

Descriptions of key issues in survey research and questionnaire design are highlighted in the following sections. Modes of data collection approaches are described together with their advantages and disadvantages. Descriptions of commonly used sampling designs are provided and the primary sources of survey error are identified. Terms relating to the topics discussed here are defined in the Research Glossary.

Survey Research

Survey research is a commonly-used method of collecting information about a population of interest. The population may be composed of a group of individuals (e.g., children under age five, kindergarteners, parents of young children) or organizations (e.g., early care and education programs, k-12 public and private schools).

There are many different types of surveys, several ways to administer them, and different methods for selecting the sample of individuals or organizations that will be invited to participate. Some surveys collect information on all members of a population and others collect data on a subset of a population. Examples of the former are the National Center for Education Statistics' Common Core of Data and the Administration for Children and Families' Survey of Early Head Start Programs (PDF).

A survey may be administered to a sample of individuals (or to the entire population) at a single point in time (cross-sectional survey), or the same survey may be administered to different samples from the population at different time points (repeat cross-sectional). Other surveys may be administered to the same sample of individuals at different time points (longitudinal survey). The Survey of Early Head Start Programs is an example of a cross-sectional survey and the National Household Education Survey Program is an example of a repeat cross-sectional survey. Examples of longitudinal surveys include the Head Start Family and Child Experiences Survey and the Early Childhood Longitudinal Study, Birth and Kindergarten Cohorts.

Regardless of the type of survey, there are two key features of survey research:

  1. Questionnaires—a predefined series of questions used to collect information from individuals.
  2. Sampling—a technique in which a subgroup of the population is selected to answer the survey questions. Depending on the sampling method, the information collected may or may not be generalized to the entire population of interest.

    The American Association for Public Opinion Research (AAPOR) offers recommendations on how to produce the best survey possible: Best Practices for Survey Research.

    AAPOR also provides guidelines on how to assess the quality of a survey: Evaluating Survey Quality in Today's Complex Environment.

Advantages and Disadvantages of Survey Research

Advantages

Disadvantages

Questionnaire Design

The two most common types of survey questions are closed-ended questions and open-ended questions.

Closed-Ended Questions

Open-Ended Questions

A well designed questionnaire is more than a collection of questions on one or more topics. When designing a questionnaire, researchers must consider a number of factors that can affect participation and the responses given by survey participants. Some of the things researchers must consider to help ensure high rates of participation and accurate survey responses include:

Questionnaires and the procedures that will be used to administer them should be pretested (or field tested) before they are used in a main study. The goal of the pretest is to identify any problems with how questions are asked, whether they are understood by individuals similar to those who will participate in the main study, and whether response options in close-ended questions are adequate. For example, a parent questionnaire that will be used in a large study of preschool-age children may be administered first to a small (often non-random) sample of parents in order to identify any problems with how questions are asked and understood and whether the response options that are offered to parents are adequate.

Based on the findings of the pretest, additions or modifications to questionnaire items and administration procedures are made prior to their use in the main study.

Resources

See the following for more information about questionnaire design:

Modes of Survey Administration

Surveys can be administered in four ways: through the mail, by telephone, in-person or online. When deciding which of these approaches to use, researchers consider: the cost of contacting the study participant and of data collection, the literacy level of participants, response rate requirements, respondent burden and convenience, the complexity of the information that is being sought and the mix of open-ended and close-ended questions.

Some of the main advantages and disadvantages of the different modes of administration are summarized below.

Mail Surveys

Telephone Surveys

In-Person Surveys

Online Surveys

Increasingly, researchers are using a mix of these methods of administration. Mixed-mode or multi-mode surveys use two or more data collection modes in order to increase survey response. Participants are given the option of choosing the mode that they prefer, rather than this being dictated by the research team. For example, the Head Start Family and Child Experience Survey (2014-2015) offers teachers the option of completing the study's teacher survey online or using a paper questionnaire. Parents can complete the parent survey online or by phone.

Resources

See the following for additional information about survey administration:

Sampling

In child care and early education research as well as research in other areas, it is often not feasible to survey all members of the population of interest. Therefore, a sample of the members of the population would be selected to represent the total population.

A primary strength of sampling is that estimates of a population's characteristics can be obtained by surveying a small proportion of the population. For example, it would not be feasible to interview all parents of preschool-age children in the U.S. in order to obtain information about their choices of child care and the reasons why they chose certain types of care as opposed to others. Thus, a sample of preschoolers' parents would be selected and interviewed, and the data they provide would be used to estimate the types of child care parents as a whole choose and their reasons for choosing these programs. There are two broad types of sampling:

Survey research studies often use a combination of these probability methods to select their samples. Multistage sampling is a probability sampling technique where sampling is carried out in several stages. It is often used to select samples when a single frame is not available to select members for a study sample. For example, there is no single list of all children enrolled in public school kindergartens across the U.S. Therefore, researchers who need a sample of kindergarten children will first select a sample of schools with kindergarten programs from a school frame (e.g., National Center for Education Statistics' Common Core of Data) (Stage 1). Lists of all kindergarten classrooms in selected schools are developed and a sample of classrooms selected in each of the sampled schools (Stage 2). Finally, lists of children in the sampled classrooms are compiled and a sample of children is selected from each of the classroom lists (Stage 3). Many of the national surveys of child care and early education (e.g., the Head Start Family and Child Experiences Survey and the Early Childhood Longitudinal Survey-Kindergarten Cohort) use a multistage approach.

Multistage, cluster and stratified sampling require that certain adjustments be made during the statistical analysis. Sampling or analysis weights are often used to account for differences in the probability of selection into the sample as well as for other factors (e.g., sampling frame, undercoverage, and nonresponse). Standard errors are calculated using methodologies that are different from those used for a simple random sample. Information on these adjustments is provided by the National Center for Education Statistics through its Distance Learning Dataset Training System.

Resources

See the following for additional information about the different types of sampling approaches and their use:

Sources of Error

Estimates of the characteristics of a population using survey data are subject to two basic sources of error: sampling error and nonsampling error. The extent to which estimates of the population mean, proportion and other population values differ from the true values of these is affected by these errors.

Measurement Error

Measurement error is the difference between the value measured in a survey or on a test and the true value in the population. Some factors that contribute to measurement error include the environment in which a survey or test is administered (e.g., administering a math test in a noisy classroom could lead children to do poorly even though they understand the material), poor measurement tools (e.g., using a tape measure that is only marked in feet to measure children's height would lead to inaccurate measurement), rater or interviewer effects (e.g., survey staff who deviate from the research protocol).

Measurement error falls into two broad categories: systematic error and random error. Systematic error is the more serious of the two.