Browse the Collection

RC Produced by Research Connections
* Peer Reviewed Journal

129 results found.

Select Citation
Result
Resource Type
*
1.

Adaptive interventions and SMART designs: Application to child behavior research in a community setting
Kidwell, Kelly M., September, 2016
American Journal of Evaluation, 37(3), 344-363

Heterogeneity between and within people necessitates the need for sequential personalized interventions to optimize individual outcomes. Personalized or adaptive interventions (AIs) are relevant for diseases and maladaptive behavioral trajectories when one intervention is not curative and success of a subsequent intervention may depend on individual characteristics or response. AIs may be applied to medical settings and to investigate best prevention, education, and community-based practices. AIs can begin with low-cost or low-burden interventions and followed with intensified or alternative interventions for those who need it most. AIs that guide practice over the course of a disease, program, or school year can be investigated through sequential multiple assignment randomized trials (SMARTs). To promote the use of SMARTs, we provide a hypothetical SMART in a Head Start program to address child behavior problems. We describe the advantages and limitations of SMARTs, particularly as they may be applied to the field of evaluation. (author abstract)

Other

get fulltext

*
2.

Analyzing early child development, influential conditions, and future impacts: Prospects of a German newborn cohort study
Weinert, Sabine, 07 October, 2016
International Journal of Child Care and Education Policy, 10(), 1-20

The paper provides an overview of a German cohort study of newborns which includes a representative sample of about 3500 infants and their mothers. The aims, challenges, and solutions concerning the large-scale assessment of early child capacities and skills as well as the measurements of learning environments that impact early developmental progress are presented and discussed. First, a brief overview of the German regulations related to early child education and care (ECEC) and parental leave as well as the study design are outlined. Then, the assessments of domain-specific and domain-general cognitive and socio-emotional indicators of early child functioning and development are described and the assessments of structural, orientational, and process quality of the children's learning environment at home and in child care are presented. Special attention is given to direct assessments and their reliability and validity; in addition, some selected results on social disparities are reported and the prospects of data analyses are discussed. (author abstract)

Other

get fulltext

*
3.

Assessing the early literacy skills of toddlers: The development of four foundational measures
Weigel, Daniel J., March/April 2017
Early Child Development and Care, 187(3-4), 744-755

Several challenges arise when researchers or practitioners attempt to assess the literacy skills of toddlers, including a lack of developmentally appropriate measures, toddlers' more limited communication ability, and how literacy is defined in the years before age three. This paper describes four new measures of early literacy development and provides preliminary evidence of their reliability and validity. Results show that the measures of Representational Knowledge, Concepts About Symbols, Book Handling Skills, and Environmental Symbols all performed well in a sample of 148 toddlers. The findings hold several implications for the study of the literacy development of toddlers. (author abstract)

Reports & Papers

*
4.

Assessing early literacy with Hispanic preschoolers: The factor structure of the Phonological Awareness Literacy Screening--Espanol
Yaden, Jr., David B., May, 2017
Hispanic Journal of Behavioral Sciences, 39(2), 193-210

For two decades, it has been recommended that assessment of literacy for preschool children be conducted in a child's primary language. However, only a few literacy assessments have been validated with a preschool, Spanish-speaking population. The purpose of the present study was to test the latent structure of the Phonological Awareness Literacy Screening for Preschool (PALS-PreK) Espanol with a sample of Spanish-speaking children in the southwestern United States. Children who could recognize at least 16 uppercase and nine lowercase letters were included in the analyses. Subscales of the PreK Espanol included Name Writing, Letter/Sound Association, Print Concepts, and Rhyme Awareness. Confirmatory factor analysis resulted in a two-factor model of alphabet knowledge (upper and lowercase letter recognition, letter sounds) and print and phonological awareness (name writing, print and rhyme awareness, letter sounds) with letter sounds loading on both factors. We found that the structure of PALS-PreK Espanol is similar to the English version and discussed implications. (author abstract)

Reports & Papers

get fulltext

5.

Assessing the implementation and cost of high quality early care and education: A review of the literature
Caronongan, Pia, April, 2016
(OPRE Report 2016-31). Washington DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

This report summarizes the findings of a literature review conducted as part of the Assessing the Implementation and Cost of High-Quality ECE (ECE-ICHQ) project funded by the Office of Planning, Research and Evaluation within the Administration for Children and Families at the U.S. Department of Health and Human Services. The project's goal is to create a technically sound and feasible instrument that will provide consistent, systematic measures of the implementation and costs of education and care in center-based settings that serve children from birth to age 5. The ultimate measures will inform research, policy, and practice by improving understanding of variations in what centers do to support quality, their associated costs, and how resources for ECE may be better aligned with expectations for quality. We reviewed the literature and research syntheses in three areas--ECE quality, implementation science, and ECE costs--to create a conceptual framework that will guide measurement development. (author abstract)

Literature Review

get fulltext

6.

Assessing the implementation and cost of high quality early care and education: A review of the literature [Executive summary]
Caronongan, Pia, April, 2016
(OPRE Report 2016-31). Washington DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

Measures of implementation and cost of ECE are needed for use and testing together with measures of quality to shed light on what it takes--in terms of activities, capacities, and money--to achieve high quality within a center. Implementation measures need to reflect what ECE centers are doing to educate and care for children and how they are doing it. Cost measures need to capture the way ECE centers allocate the resources they have to work with. The Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (DHHS) launched a new project--Assessing the Implementation and Cost of High-Quality ECE (ECE-ICHQ)--to develop measures of implementation and cost for an ECE center. The measures will inform research, policy, and practice by improving understanding of variations in what centers do to support quality, their associated costs, and how resources for ECE may be better aligned with expectations for quality. The project's goal is to create a technically sound, feasible, and useful instrument that will provide consistent, systematic measures of the implementation and costs of education and care in center-based settings that serve children from birth to age 5 ("ECE centers"). By "costs" we mean how much it costs to operate a center, including costs incurred by the center, as well as the value of in-kind contributions such as space or labor. (author abstract)

Executive Summary

get fulltext

*
7.

Assessing quality of kindergarten classrooms in Singapore: Psychometric properties of the Early Childhood Environment Rating Scale--Revised
Bull, Rebecca, April, 2017
International Journal of Early Childhood, 49(1), 1-20

The early childhood sector in Singapore has witnessed vast changes in the past two decades. One of the key policy aims is to improve classroom quality. To ensure a rigorous evaluation of the quality of early childhood environments in Singapore, it is important to determine whether commonly used assessments of quality are valid indicators across different national and community contexts. This study investigated the validity of the Early Childhood Environment Rating Scale--Revised (ECERS-R) in Singapore focused on three major aspects: response process, structural, and criterion validity. Data were collected from 108 classrooms of the main providers of kindergarten programs in Singapore. Initial analyses showed that the quality indicators in most of the ECERS-R items were disordered, which indicated problems with the hypothesized response process underlying the standard ECERS-R scoring system in which high-level indicators are only rated if low-level indicators are met (stop-scoring). To deal with the problem of the "stop-scoring" method, we used the data from a full-scoring method (in which every indicator on every item was rated) in subsequent factor analyses to investigate the internal structure of the ECERS-R. Twenty-two of the 34 items were retained and found to load on two factors, one relating to activities/materials and the other related to language/interactions. Both of the identified subscales and the combined scale showed good internal consistency. The factor score on the combined scale mirrored the ECERS-R full score and could discriminate between classrooms in the top and bottom quartiles of quality scores. Findings on the relationship between the identified factors to the teacher-child ratio and to an alternative measure of classroom quality provided further evidence for criterion validity of the ECERS-R. Implications for the future development and use of the ECERS-R are discussed. (author abstract)

Reports & Papers

get fulltext

8.

Best practices in creating and adapting quality rating and improvement system (QRIS) rating scales
Burchinal, Margaret, May, 2016
(OPRE Research Brief 2016-25). Washington DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The brief summarizes an analysis that uses the data from six large studies of early care and education to simulate state QRIS ratings. The results suggest that QRIS ratings can achieve their desired goal of predicting gains in child outcomes when attention is paid to the psychometric principles of scale development including: dimensionality (ensuring that a scale represents one, not multiple dimensions), selecting items with strong evidence, and scoring items using established criteria for cut points. (author abstract)

Fact Sheets & Briefs

get fulltext

*
9.

Between deposit and release: Research Connections and your data
Bleckman, Johanna, 18 August, 2016
New York: Child Care & Early Education Research Connections

One way that Research Connections promotes secondary research and policy making is by performing data processing and enhancement on deposited research data in order to add value to data by making them easier to use for secondary analysis. The purpose of this video is to describe the process of archiving data with Research Connections and understand Research Connections' role in archiving your data. Viewers will: (1) Discover the benefits of contributing your research data to the Research Connections data archive. (2) Understand the ICPSR "pipeline" for processing and enhancement and the steps that Research Connections performs on your data before releasing a study on the Research Connections website. (3) Learn data management tips and techniques that can be incorporated into your research project to facilitate a smooth transition to the Research Connections archive at project completion.

Multimedia

view multimedia

*
10.

Between deposit and release: Research Connections and your data [PowerPoint]
Bleckman, Johanna, 18 August, 2016
New York: Child Care & Early Education Research Connections

This PowerPoint presentation accompanies a webinar that describes the process of archiving data with Research Connections. Viewers will: (1) Discover the benefits of contributing your research data to the Research Connections data archive. (2) Understand the ICPSR "pipeline" for processing and enhancement and the steps that Research Connections performs on your data before releasing a study on the Research Connections website. (3) Learn data management tips and techniques that can be incorporated into your research project to facilitate a smooth transition to the Research Connections archive at project completion.

Other

get fulltext

11.

A brief description of the Examining Data Informing Teaching (EDIT) measure
Akers, Lauren, November, 2016
(OPRE Brief No. 2016-104). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

This brief introduces the Examining Data Informing Teaching (EDIT) measure for researchers who seek to learn about preschool classroom teachers' use of ongoing assessment data to individualize instruction. In this brief, we discuss the EDIT's multi-method procedures, structure, and scoring; testing to date and future testing needs; a process for training EDIT raters; and potential uses of the measure. (author abstract)

Fact Sheets & Briefs

get fulltext

12.

Brigance Early Childhood Screens III
Brigance, Albert H., 2013
North Billerica, MA: Curriculum Associates.

Instruments

13.

Caring for Our Children Basics Health and Safety Standards Alignment Tool for Child Care Centers and Family Child Care Homes
National Center on Early Childhood Quality Assurance, June, 2016
Fairfax, VA: National Center on Early Childhood Quality Assurance.

Caring for Our Children Basics (CFOCB) represents the minimum health and safety standards experts believe should be in place where children are cared for outside their own homes, whether in a home-based program or center-based facility. It does not, however, represent all standards that should be present to achieve the highest quality of care and early learning. For example, the caregiver training requirements outlined in these standards are designed only to prevent harm to children, not to ensure children's optimal development and learning. Although use of Caring for Our Children Basics is voluntary, the Administration for Children and Families (ACF) hopes Caring for Our Children Basics will be a helpful resource for States and other entities as they work to improve health and safety standards in both licensing and quality rating improvement systems (QRIS). This tool provides a simple format for States and Territories to compare their current early childhood program requirements and standards against the recommended health and safety standards in CFOCB. (author abstract)

Instruments

get fulltext

*
14.

The Child-care Food and Activity Practices Questionnaire (CFAPQ): Development and first validation steps
Gubbels, Jessica S., August, 2016
Public Health Nutrition, 19(11), 1964-1975

Objective: To develop and validate a questionnaire to measure food-related and activity-related practices of child-care staff, based on existing, validated parenting practices questionnaires. Design: A selection of items from the Comprehensive Feeding Practices Questionnaire (CFPQ) and the Preschooler Physical Activity Parenting Practices (PPAPP) questionnaire was made to include items most suitable for the child-care setting. The converted questionnaire was pre-tested among child-care staff during cognitive interviews and pilot-tested among a larger sample of child-care staff. Factor analyses with Varimax rotation and internal consistencies were used to examine the scales. Spearman correlations, t tests and ANOVA were used to examine associations between the scales and staff's background characteristics (e.g. years of experience, gender). Setting: Child-care centres in the Netherlands. Subjects: The qualitative pre-test included ten child-care staff members. The quantitative pilot test included 178 child-care staff members. Results: The new questionnaire, the Child-care Food and Activity Practices Questionnaire (CFAPQ), consists of sixty-three items (forty food-related and twenty-three activity-related items), divided over twelve scales (seven food-related and five activity-related scales). The CFAPQ scales are to a large extent similar to the original CFPQ and PPAPP scales. The CFAPQ scales show sufficient internal consistency with Cronbach's [alpha] ranging between 0.53 and 0.96, and average corrected item-total correlations within acceptable ranges (0.30-0.89). Several of the scales were significantly associated with child-care staff's background characteristics. Conclusions: Scale psychometrics of the CFAPQ indicate it is a valid questionnaire that assesses child-care staff's practices related to both food and activities. (author abstract)

Reports & Papers

get fulltext

*
15.

Comparing inference approaches for RD designs: A reexamination of the effect of Head Start on child mortality
Cattaneo, Matias D., Summer 2017
Journal of Policy Analysis and Management, 36(3), 643-681

The regression discontinuity (RD) design is a popular quasi-experimental design for causal inference and policy evaluation. The most common inference approaches in RD designs employ "flexible" parametric and nonparametric local polynomial methods, which rely on extrapolation and large-sample approximations of conditional expectations using observations somewhat near the cutoff that determines treatment assignment. An alternative inference approach employs the idea of local randomization, where the very few units closest to the cutoff are regarded as randomly assigned to treatment and finite-sample exact inference methods are used. In this paper, we contrast these approaches empirically by re-analyzing the influential findings of Ludwig and Miller (2007), who studied the effect of Head Start assistance on child mortality employing parametric RD methods. We first review methods based on approximations of conditional expectations, which are relatively well developed in the literature, and then present new methods based on randomization inference. In particular, we extend the local randomization framework to allow for parametric adjustments of the potential outcomes; our extended framework substantially relaxes strong assumptions in prior literature and better resembles other RD inference methods. We compare all these methods formally, focusing on both estimands and inference properties. In addition, we develop new approaches for randomization-based sensitivity analysis specifically tailored to RD designs. Applying all these methods to the Head Start data, we find that the original RD treatment effect reported in the literature is quite stable and robust, an empirical finding that enhances the credibility of the original result. All the empirical methods we discuss are readily available in general purpose software in R and Stata; we also provide the dataset and software code needed to replicate all our results. (author abstract)

Reports & Papers

get fulltext

*
16.

Considerations for time sampling interval durations in the measurement of young children's classroom engagement
Zakszeski, Brittany N., May, 2017
Topics in Early Childhood Special Education, 37(1), 42-53

Classroom engagement is important to young children's academic and social development. Accurate methods of capturing this behavior are needed to inform and evaluate intervention efforts. This study compared the accuracy of interval durations (i.e., 5 s, 10 s, 15 s, 20 s, 30 s, and 60 s) of momentary time sampling (MTS) in approximating the duration of classroom engagement as measured by continuous duration recording (CDR). Twenty-four sessions of children in large-group instruction were observed using the Behavioral Observation of Students in School for Early Education (BOSS-EE). In general, shorter intervals produced engagement estimates that more highly correlated with CDR data and had less measurement error relative to estimates from longer intervals. Findings suggest that selection of MTS interval duration may affect the accuracy of estimates for children with low levels of engagement to a greater extent relative to children with high levels of engagement. Implications and future directions are discussed. (author abstract)

Reports & Papers

get fulltext

17.

Considerations in preparing to analyze administrative data to address child care and early education research questions
Lin, Van-Kim Bui, February, 2017
(OPRE Research Brief No. 2017-18). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The purpose of this resource is to help researchers prepare for issues that may arise when using administrative data as the primary data source for a research project. This is the third in a series of resources related to the analysis of administrative data. The first resource, Developing Collaborative Partnerships with State Agencies to Strengthen Research Using Early Care and Education Administrative Data, provides considerations for building a strong partnership between researchers who want to analyze administrative data and the state partners who oversee the administrative data. The second resource, Determining the Feasibility of Using State Early Care and Education Administrative Data, is designed to help researchers and their state partners determine whether analyzing administrative data is feasible and appropriate for addressing their child care and early education research questions. Once researchers and state agency partners have determined that it is feasible to use administrative data to address a question of shared interest, then this third resource can be helpful in preparing to analyze the data. These resources have been designed for use by researchers who are new to the analysis of administrative data as well as seasoned researchers who are expanding their research to include new types of administrative data or expanding into new states or new agencies. The information generated for each of these resources was developed through conversations with grantees and researchers who have experience analyzing state administrative data. This resource is organized into six sections applicable to analyses of administrative data: 1) understanding the scope and limitations of administrative data when developing an analysis plan, 2) selecting variables to analyze, 3) assessing the feasibility of the plan, 4) preparing a data request, 5) creating a dataset, and 6) developing and maintaining adequate data documentation. For each of the sections, we have provided considerations, examples, and/or questions to ask that are specific to the use of state administrative data related to child care and early education. The purpose of each section is to provide insights to help researchers in identifying variables and problem solving issues that may arise in the analysis of administrative datasets. Although the sections are described separately, we expect the process to be iterative rather than linear, and to require continued discussions and reconsideration of decisions as new information is learned. (author abstract)

Other

get fulltext

*
18.

Constructs of quality in early childhood education and care: A close examination of the NQS Assessment and Rating Instrument
Jackson, Jen, September, 2015
Australasian Journal of Early Childhood, 40(3), 46-50

The National Quality Standard (NQS) Assessment and Rating Instrument is an important tool for supporting reliability and validity in the assessment and rating process for Australian early childhood education and care services. The Instrument provides a guide for authorised officers and practitioners, by describing each element of the NQS at each of the three most frequently applied quality levels: Working Towards NQS, Meeting NQS and Exceeding NQS. This minor research project involved detailed content analysis of the Instrument, to identify the constructs used to differentiate between the three quality levels and the frequency with which each construct occurred. The analysis found that intentionality, frequency, extent and inclusivity are the four constructs most commonly used for differentiating between NQS quality levels. (author abstract)

Reports & Papers

*
19.

Conversation Compass(C) Communication Screener: A conversation screener for teachers
Gardner, Shari L., March/April 2017
Early Child Development and Care, 187(3-4), 487-497

The purpose of this study was to report preliminary reliability and validity data from the Conversation Compass(C) Communication Screener (CCCS), a teacher-reported language screener intended to capture children's skills related to classroom conversations with peers and teachers. Three preschool teachers completed the CCCS and the Child Observation Record (COR) for 36 students. Results indicated six subscales -- literacy, clarity, social communication, decontextualized thinking, grammar, and negative communication behaviours -- in the CCCS were reliable at Cronbach's alphas of .71 or greater. Results also indicated its concurrent validity with the COR. Lastly, analyses indicated the screener was sensitive to maturation in conversation skills in that the measure was correlated with age. Implications are discussed in relation to how teachers can use this tool to inform their classroom practices. (author abstract)

Reports & Papers

20.

COR Advantage
High/Scope Educational Research Foundation, 2014
Ypsalanti, MI: High/Scope Educational Research Foundation

Instruments

get fulltext

*
21.

Creating spaces to hear parents' voices: Methodological reflections on the Families Commission's early childhood care and education project involving some migrant and former refugee families
Kindon, Sara, June, 2009
Social Policy Journal of New Zealand, 35, 139-151

How do recently arrived migrant and former refugee families from non-English-speaking backgrounds in Aotearoa New Zealand balance work, study and childcare? How do they access and experience early childhood care and education? This paper describes and reflects on a Families Commission-funded qualitative research project which sought to generate answers to these questions via focus groups and participatory diagramming. It outlines the context within which the research was commissioned before discussing the rationale and approach adopted. It offers reflections on the lessons learnt from negotiating cultural, linguistic and contextual differences, and from attempting to create appropriate spaces in which to listen to parents' experiences, including the context of the accountability environment of a New Zealand Crown entity. (author abstract)

Other

*
22.

A critical review of the advantages and limitations of using large-scale national surveys to examine childcare patterns and the ECEC workforce in Britain
Simon, Antonia, 2017
Early Years: An International Journal of Research and Development, , 1-13

OECD countries have established statistical collections to ensure quality within Early Childhood Education and Care (ECEC). Focusing on one part of ECEC -- preschool 'childcare services' -- this paper critically reviews statistical collections specifically designed to measure childcare patterns in England alongside UK data collected for other purposes which can be used to examine childcare patterns. The paper evaluates how far these data provide a reliable basis for examining the childcare workforce, how well childcare usage and provision patterns can be analysed and the degree to which the data provide comparable geographical coverage. Results show analysis is restricted by the various ways data-sets count and classify occupations. Differences in geographical coverage make them difficult to compare. More refinement of occupation categories would make existing sources more useful. The themes discussed here are relevant for other countries seeking to understand how best to utilise their statistical collections for examining childcare patterns. (author abstract)

Reports & Papers

23.

Data collection and use: An early childhood perspective
Bourassa, Jacqueline, 13 August, 2015
Washington, DC: Regional Educational Laboratory Northeast & Islands

This webinar presents findings from a study that explores the collection and use of data in several New England preschool programs. Analysis focuses on three domains: early learning outcomes (math, reading, and social-emotional); classroom quality measures; and dosage. The report addresses the challenges of combining multiple sources of data, as well as advising awareness of multiple explanations that can occur within data analysis. Presenters focus on data collection and use at both state and program levels.

Multimedia

view multimedia

24.

Data collection and use: An early childhood perspective [PowerPoint]
Bourassa, Jacqueline, 13 August, 2015
Washington, DC: Regional Educational Laboratory Northeast & Islands

This PowerPoint presentation accompanies a webinar that presents findings from a study which explores the collection and use of data in several New England preschool programs. Analysis focuses on three domains: early learning outcomes (math, reading, and social-emotional); classroom quality measures; and dosage. The report addresses the challenges of combining multiple sources of data, as well as advising awareness of multiple explanations that can occur within data analysis. Presenters focus on data collection and use at both state and program levels.

Other

get fulltext

25.

Design Options for Understanding Child Maltreatment
Stagner, Matthew, 2015
Mathematica Policy Research

In September 2015, the Office of Planning, Research and Evaluation (OPRE) in collaboration with the Children's Bureau awarded a contract to Mathematica Policy Research to carry out the Design Options for Understanding Child Maltreatment Incidence. The purpose of this project is to develop design options for a potential future study or group of studies that would leverage existing administrative data, innovative methods, and advanced statistical techniques to obtain accurate and ongoing surveillance on both the incidence of child abuse and neglect and types of related risk. This project will allow for the identification and prioritization of key research questions; exploration of design options, including innovative methodological approaches; review of existing administrative datasets and ongoing surveys; examination of measurement issues; and consideration of resource allocation.

Administration for Children and Families/OPRE Projects

26.

Determining the feasibility of using state early care and education administrative data
Lin, Van-Kim Bui, February, 2017
(OPRE Research Brief No. 2017-17). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The purpose of this resource is to help early childhood researchers determine the feasibility of using administrative data in their research. Administrative data refers to information about individual children, families, and/or service providers that is collected and maintained as a part of program operations. Administrative data can describe and inform the implementation of policies and programs. This resource provides questions for researchers to consider and discuss with agency leaders before finalizing a research plan that uses administrative data. This resource has been designed for use by researchers who are new to the analysis of administrative data, as well as seasoned users of administrative data who are expanding their research to include new types of administrative data (e.g., expanding to a new state or new agency). The questions generated for this resource were developed through conversations with grantees and researchers who had experience analyzing state administrative data. Two other resources may be helpful to researchers interested in using early care and education administrative data: Developing Collaborative Partnerships with State Agencies to Strengthen Research Using Early Care and Education Administrative Data and Considerations in Preparing to Analyze Administrative Data to Address Early Care and Education Related Research Questions. We have organized this resource into three sections, each of which covers a different set of questions to help determine the feasibility of using administrative data: 1) data stewardship and management, 2) data contacts and coordination, and 3) data usability. (author abstract)

Other

get fulltext

27.

Developing collaborative partnerships with state agencies to strengthen research using early care and education administrative data
Maxwell, Kelly, February, 2017
(OPRE Research Brief No. 2017-16). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

Collaborative partnerships between researchers and agency staff are mutually beneficial, and they respect and build upon the specific expertise of each partner. While collaborative partnerships may take time to develop, they not only improve the feasibility and quality of research using administrative data but also support the applicability of research to inform policy and practice. Working together, researchers and agency staff can co-construct research questions that address issues related to program operations, policies, or pressing issues in the field that can be adequately answered with administrative data. The purpose of this resource is to offer ideas to researchers about how to build relationships with state partners to facilitate the effective use of administrative data for research and to inform policy. Different contexts, histories, and institutional capacities require distinct approaches to collaboration, so we offer researchers a range of possible strategies for establishing a partnership with state agency staff. This is the first in a series of three resources designed to help researchers interested in using administrative data. The other briefs in this series are entitled Determining the Feasibility of Using State Administrative Data and Considerations in Preparing to Analyze Administrative Data to Address Early Care and Education Related Research Questions. (author abstract)

Other

get fulltext

28.

Developing a tool to examine teachers' use of ongoing child assessment to individualize instruction
Monahan, Shannon, November, 2016
(OPRE Report No. 2016-103). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

In 2012, the Office of Planning, Research, and Evaluation at the Administration for Children and Families (ACF) engaged Mathematica Policy Research and its partners to conduct a project titled "Assessing Early Childhood Teachers' Use of Child Progress Monitoring to Individualize Teaching Practices." The purpose of the project was twofold: (1) to develop a research-informed conceptual model for early childhood teachers' use of ongoing assessment to individualize instruction, and (2) to create a measure to examine this process. Prior reports describe in detail the results of a literature review, conceptual framework, and measurement plan (Akers et al. 2014; Atkins-Burnett et al. 2014). This report describes the iterative development of the Examining Data Informing Teaching (EDIT) measure. This report includes the results of a pretest study in 18 classrooms and a proposal for next steps for the EDIT. (author abstract)

Reports & Papers

get fulltext

29.

Developing a tool to examine teachers' use of ongoing child assessment to individualize instruction [Executive summary]
Monahan, Shannon, November, 2016
(OPRE Report No. 2016-103). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

In 2012, the Office of Planning, Research, and Evaluation at the Administration for Children and Families (ACF) engaged Mathematica Policy Research and its partners to conduct a project titled "Assessing Early Childhood Teachers' Use of Child Progress Monitoring to Individualize Teaching Practices." The purpose of the project was twofold: (1) to develop a research-informed conceptual model for early childhood teachers' use of ongoing assessment to individualize instruction, and (2) to create a measure to examine this process. Prior reports describe in detail the results of a literature review, conceptual framework, and measurement plan (Akers et al. 2014; Atkins-Burnett et al. 2014). This report describes the iterative development of the Examining Data Informing Teaching (EDIT) measure. This report includes the results of a pretest study in 18 classrooms and a proposal for next steps for the EDIT. (author abstract)

Executive Summary

get fulltext

*
30.

Development and transfer of vocabulary knowledge in Spanish-speaking language minority preschool children
Goodrich, J. Marc, September, 2016
Journal of Child Language, 43(5), 969-992

In this study we evaluated the predictive validity of conceptual scoring. Two independent samples of Spanish-speaking language minority preschoolers (Sample 1: N= 96, mean age = 54.51 months, 54.3% male; Sample 2: N= 116, mean age = 60.70 months, 56.0% male) completed measures of receptive, expressive, and definitional vocabulary in their first (L1) and second (L2) languages at two time points approximately 9-12 months apart. We examined whether unique L1 and L2 vocabulary at time 1 predicted later L2 and L1 vocabulary, respectively. Results indicated that unique L1 vocabulary did not predict later L2 vocabulary after controlling for initial L2 vocabulary. An identical pattern of results emerged for L1 vocabulary outcomes. We also examined whether children acquired translational equivalents for words known in one language but not the other. Results indicated that children acquired translational equivalents, providing partial support for the transfer of vocabulary knowledge across languages. (author abstract)

Reports & Papers

get fulltext

*
31.

The development and validation of the Behavior and Emotion Expression Observation System to characterize preschoolers' social and emotional interactions
Johnson, Stacy R., October, 2016
Early Education and Development, 27(7), 896-913

Research Findings: This article describes the development and evaluation of the Behavior and Emotion Expression Observation System (BEEOS), a direct observation tool to characterize preschoolers' social and emotion behaviors during semistructured activities in the classroom. The BEEOS was used to observe 148 Head Start preschoolers, and questionnaires were completed by teachers and parents to provide psychometric comparisons to the observational system. Findings support both the reliability and criterion-related validity of the behavior indicators and emotion expressions that make up the BEEOS. Preschoolers in the current sample engaged in expected patterns of social behavior and emotion expression. Furthermore, both constructs were related to teacher and parent reports of preschoolers' social behavior, externalizing and internalizing symptomatology, emotion regulation, and perceived emotion expressions. Practice or Policy: Findings suggest that the BEEOS can be used to supplement caregiver report and standardized assessments of preschoolers' social and emotion behaviors. The potential applications and implications of the BEEOS for research and education are discussed and include informing teaching planning and practices as well as the effectiveness of social-emotional interventions. (author abstract)

Reports & Papers

get fulltext

32.

Doing something differently: Lessons learned from a case study using implementation science to guide program change
Mathias, Debi, 12 December, 2016
Boston, MA: Build Initiative

This webinar discusses implementation science and how it can guide quality improvement initiatives and inform technical assistance. Presenters explore the role of implementation teams, data and feedback loops, and infrastructure as core elements that support continuous quality improvement. A case study of Rowan University's Early Childhood Demonstration Center in New Jersey provides an example of how these core elements are incorporated within stages of implementation. Implications for program administrators and technical assistants are discussed.

Multimedia

view multimedia

33.

Doing something differently: Lessons learned from a case study using implementation science to guide program change [PowerPoint]
Mathias, Debi, 12 December, 2016
Boston, MA: Build Initiative

This PowerPoint presentation accompanies a webinar that discusses implementation science and how it can guide quality improvement initiatives and inform technical assistance. Presenters explore the role of implementation teams, data and feedback loops, and infrastructure as core elements that support continuous quality improvement. A case study of Rowan University's Early Childhood Demonstration Center in New Jersey provides an example of how these core elements are incorporated within stages of implementation. Implications for program administrators and technical assistants are discussed.

Other

get fulltext

34.

Early care, early education, and home visiting in American Indian and Alaska Native communities: Design options for assessing early childhood needs
Malone, Lizabeth M., April, 2016
(OPRE Report 2016-49). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The report documents the process of creating three design topics for an early childhood needs assessment of American Indian and Alaska Native (AI/AN) children and families. The Administration for Children and Families (ACF) developed three broad design topics, and Mathematica Policy Research convened a community of learning (CoL) made up of child care practitioners and researchers, Head Start/Early Head Start practitioners and researchers, tribal home visiting practitioners and researchers, ACF federal staff, including representatives from the Office of Child Care, the Office of Head Start, the Office of Early Childhood Development, and the Office of Planning, Research and Evaluation, and research partners from the Tribal Early Childhood Research Center. The CoL met monthly throughout 2015 and provided a range of perspectives on the key decision points about the designs that came up in those discussions. The three design topics are as follows: Design One will describe the population of AI/AN children and families and their participation in early childhood services based on existing data sources. To the extent possible, this design will provide a broad picture of the programs and providers serving AI/AN children and families at a national level. Design Two will study service organization and delivery systems in AI/AN communities, including the current number of children served and not served, workforce capacity, and cultural resources at the community level and will involve new data collection. Design Three will assess key features needed to support AI/AN communities' capacity for conducting early childhood needs assessments at the community level and will involve new data collection. The report begins with a description of the framework underlying each design topic: the population of interest and the definition of early childhood needs, services, and indicators, followed by details on each of the three design topics. Each chapter addresses the key research questions for the design topic, the population of interest, measurement topics to consider when addressing the research questions, and data sources, including primary data collection or existing data sources available for secondary analysis. The report concludes with a summary of each design topic and future considerations. The goal of this report is to inform the future design of a needs assessment. However, it does not include the details for specific sample designs, data collection protocols or instruments, or analysis plans. (author abstract)

Other

get fulltext

35.

Early Head Start Research and Evaluation Project (EHSREP): 1996-2010 measures compendium
Klein, Ashley Kopack, November, 2016
(OPRE Report 2016-101). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

Early Head Start (EHS) is a two-generation program for pregnant women and families with infants or toddlers. Offered to those with limited incomes, its goal is to enhance children's development and health and to strengthen family and community partnerships. The Early Head Start Research and Evaluation Project (EHSREP), sponsored by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), was designed to answer questions about the overall impact of EHS programs and services on children and families and reveal how specific types of programs and services affect children and families that have different characteristics and life circumstances. Mathematica Policy Research led the rigorous evaluation, which was launched at about the same time the EHS program was authorized in 1995. In this data compendium, we provide a single source for information about the measures used throughout the EHSREP. We begin with an overview of the EHSREP design and then report the sample, data collection instruments, and response rates for each of three EHSREP data collection phases (birth to age 3, prekindergarten, and grade 5). Next, we describe the various data sets and documentation that are available to data users, and we provide a general description of how we have organized the more detailed information on the measures we used to create variables and scores for the public use data files. Appendix A contains detailed descriptions of the measures, including measure citations, publisher psychometrics and permissions, the wave or waves in which each measure was used, and information on the scales or variables in the EHSREP data set that were derived from these measures. (author abstract)

Other

get fulltext

36.

Early Learning Mentor Coach (ELMC) Study: Coach Survey
Howard, Eboni C., 11 June, 2011
Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

Instruments

get fulltext

37.

Early Screening Project: A proven child find process
Lane, Kathleen L., 2011
In Lane, K. L., Menzies, H. M., & Oakes, W. P., Systematic screenings of behavior to support instruction: From preschool to high school (pp. 57-93). New York: Guilford Press

In this chapter, we begin with an overview of the ESP, including step-by-step directions on how to complete this multistage assessment, and provide responses to frequently asked questions regarding logistical considerations. Next, we synthesize the supporting research conducted with the ESP, including studies (1) examining the reliability and validity of this screening tool as well as (2) demonstrating how to use the ESP to identify students who might require secondary (Tier 2) or tertiary (Tier 3) supports. Then we provide a balanced discussion of the strengths and challenges associated with preparation, administration, scoring, and interpretation. We conclude this chapter with two illustrations. The first explains how to monitor the overall level of risk and then use that information to provide individual supports in a large preschool setting. The second illustration demonstrates how to use the data from the screening tool in a three-tiered model of support in an elementary school kindergarten program to identify students for a manualized tertiary support, First Step to Success (Walker et al., 1997). (author abstract)

Other

*
38.

An early years toolbox for assessing early executive function, language, self-regulation, and social development: Validity, reliability, and preliminary norms
Howard, Steven J., June, 2017
Journal of Psychoeducational Assessment, 35(3), 255-275

Several methods of assessing executive function (EF), self-regulation, language development, and social development in young children have been developed over previous decades. Yet new technologies make available methods of assessment not previously considered. In resolving conceptual and pragmatic limitations of existing tools, the Early Years Toolbox (EYT) offers substantial advantages for early assessment of language, EF, self-regulation, and social development. In the current study, results of our large-scale administration of this toolbox to 1,764 preschool and early primary school students indicated very good reliability, convergent validity with existing measures, and developmental sensitivity. Results were also suggestive of better capture of children's emerging abilities relative to comparison measures. Preliminary norms are presented, showing a clear developmental trajectory across half-year age groups. The accessibility of the EYT, as well as its advantages over existing measures, offers considerably enhanced opportunities for objective measurement of young children's abilities to enable research and educational applications. (author abstract)

Reports & Papers

get fulltext

*
39.

Ecomapping as a research tool for informal child care
Johnson, Cleo Jacobs, August, 2017
Early Education and Development, 28(6), 705-714

Research Findings: Many young children spend crucial developmental years in informal, home-based child care (HBCC) settings, but parents and others share concerns about HBCC quality. We applied the ecomap method in a descriptive study of racially, ethnically, and linguistically diverse informal caregivers and parents to capture their informal caregiving arrangements, social networks, and social supports. For the parents in our study, informal child care was a flexible, affordable, and readily accessible child care option. For caregivers, informal care provided opportunities to help family and friends and to earn extra income. Social networks were characterized largely by relationships with family and friends, and social supports tended to be strong and mutually beneficial. Practice or Policy: We found that ecomapping is an engaging and flexible method of capturing and understanding complex child care arrangements and social networks in HBCC settings. As a research tool, ecomapping can help improve the quality of care in informal settings by identifying and meeting the needs of parents and HBCC providers and informing program development and delivery strategies. (author abstract)

Reports & Papers

get fulltext

40.

Essential organizational supports for early education: The development of a new survey tool to measure organizational conditions
Ehrlich, Stacy, July, 2016
Chicago: University of Chicago Consortium on School Research.

We present this brief to practitioners and researchers who are interested in measuring the quality of organization-level supports. For all audiences, this brief aims to explain why the early education field would benefit from a measurement system that captures the strength of organizational processes. By providing concrete definitions of the organizational conditions being measured by our new surveys, we encourage practitioners to begin conceptualizing what this may mean for their own work. For other researchers seeking to develop new surveys, this brief provides a roadmap of our rigorous survey development process, whereby we describe our methods for achieving reliable and valid measurement of our intended constructs. (author abstract)

Reports & Papers

get fulltext

41.

Estimating the cost of raising child care workers' wages for state subsidy programs: A methodology applied to California's new state minimum wage law
Thomason, Sarah, December, 2016
Berkeley, CA: University of California, Berkeley, Center for Labor Research and Education.

In April 2016, California passed legislation to increase the state minimum wage annually until it reaches $15 an hour in 2023 for all businesses. As a result, child care centers and licensed in-home providers will be required to increase the wages of their employees who currently earn less than the new minimum wage. Because a large proportion of workers in the child care industry is low-wage, this could have a significant impact on providers. Providers with private clients may respond by raising their prices to cover the cost of the wage increase. However, the amount providers receive for caring for children covered by state child care subsidy programs is determined by state and county reimbursement rates. Without the ability to change the amount charged for caring for subsidized children, child care centers or licensed in-home facilities may not be able to cover the cost of raising workers' wages to the new minimum wage. In this memo, we describe a methodology we have developed for estimating the additional child care subsidy funding needed to cover the cost of a state minimum wage increase for programs administered by the California Department of Education (CDE) and the Department of Social Services through the CalWORKs 1 (Welfare to Work) program. (author abstract)

Other

get fulltext

*
42.

Evaluating the validity of classroom observations in the Head Start Designation Renewal System
Mashburn, Andrew J., January-March 2017
Educational Psychologist, 52(1), 38-49

Classroom observations are increasingly common in education policies as a means to assess the quality of teachers and/or education programs for purposes of making high-stakes decisions. This article considers one policy, the Head Start Designation Renewal System (DRS), which involves classroom observations to assess the quality of Head Start programs in order to decide whether their funding is renewed. This article applies an argument-based approach for evaluating the validity of observational assessments that (a) explicates assumptions that underlie the presumed logic, leading from the collection of scores from observations of Head Start classrooms, to the inference that scores assess the quality of Head Start programs, to the decision to renew funding to Head Start programs, and (b) summarizes evidence that speaks to the plausibility of each assumption. There was limited evidence to support the plausibility of many assumptions, including those pertaining to score generalizability, predictive validity, and the cutoff scores set as minimum standards of quality. Implications for improving the validity of classroom observations and the accuracy and fairness of decisions in the Head Start DRS are discussed. (author abstract)

Other

get fulltext

*
43.

Examining a self-report measure of parent-teacher cocaring relationships and associations with parental involvement
Lang, Sarah N., January, 2017
Early Education and Development, 28(1), 96-114

By adapting a self-administered assessment of coparenting, we sought to provide a new tool, the Cocaring Relationship Questionnaire, to measure parent-teacher, or cocaring relationships, and provide additional construct validity for the multidimensional concept of cocaring. Next, recognizing the importance of parental involvement for young children's development, we examined the associations between dimensions of cocaring and aspects of parental involvement. We investigated the parent-teacher relationships of 90 families utilizing full-time, center-based childcare for their 12-36 month old children. Parents and teachers completed a set of questionnaires. Research findings: exploratory factor analysis revealed a four factor structure for the cocaring relationship: Support, Undermining, Endorsement, and Agreement. After controlling for a number of child- and parent-level covariates, parents' perceptions of different dimensions within the cocaring relationship were associated with different aspects of their self-reported and teacher-reported involvement. Most notably, parents' perception of cocaring support was positively associated with three different forms of parental involvement. Practice or Policy: the Cocaring Relationship Questionnaire offers researchers and practitioners a means to assess multiple dimensions within parent-teacher relationships. Understanding that parent-teacher relationships are multifaceted can help practitioners consider their interactions with families in new ways, which may influence, or be influenced by, parental involvement. (author abstract)

Reports & Papers

get fulltext

*
44.

Exploring the use of emoji as a visual research method for eliciting young children's voices in childhood research
Fane, Jennifer, 2016
Early Child Development and Care, , 1-16

Recognition of the need to move from research on children to research with children has prompted significant theoretical and methodological debate as to how young children can be positioned as active participants in the research process. Visual research methods such as drawing, photography, and videography have received substantive attention in child-centred research paradigms. However, despite their increasing ubiquity in young children's lifeworlds, technology or media-based visual materials have received little interest. This article reports on a study which used emoji as a visual research method for eliciting young children's (aged three to five years) understandings and experiences of well-being. Findings elucidate the capacity of emoji as a visual research method for eliciting children's voices, and considerations for its use in child research. (author abstract)

Reports & Papers

*
45.

Factor analysis of the Preschool Behavioral and Emotional Rating Scale for children in Head Start programs
Cress, Cynthia J., August, 2016
Journal of Psychoeducational Assessment, 34(5), 473-486

Strength-based assessment of behaviors in preschool children provides evidence of emotional and behavioral skills in children, rather than focusing primarily on weaknesses identified by deficit-based assessments. The Preschool Behavioral and Emotional Rating Scales (PreBERS) is a normative assessment of emotional and behavioral strengths in preschool children. The PreBERS has well-established reliability and validity for typically developing children as well as children with identified special education needs, but this has not yet been established for children in Head Start programs, who tend to be at high risk for development of emotional and behavioral concerns. This study explores the factorial validity of the PreBERS scores for a large sample of children participating in Head Start programs around the United States. Results not only confirm the fit of the four-factor model of the PreBERS for this population, but also demonstrate the application of a bifactor model to the structure of the PreBERS which, in turn, allows for the computation of model-based reliability estimates for the four subscales (Emotional Regulation, School Readiness, Social Confidence, Family Involvement) and overall strength index score. The implications suggest that the PreBERS items are reliable scores that can be used to identify behavioral strengths in preschool children in Head Start, and support planning of interventions to selectively address component skills to promote child social and academic success. (author abstract)

Reports & Papers

get fulltext

46.

Family and Provider/Teacher Relationship Quality measures: Updated user's manual
Kim, Kwang, April, 2015
(OPRE Report 2014-65). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The Family and Provider/Teacher Relationship Quality (FPTRQ) project, sponsored by the Administration for Children and Families' Office of Head Start (OHS) and Office of Planning, Research and Evaluation (OPRE), was a four-year effort to develop measures of the quality of family and provider/teacher relationships in early care and education (ECE) settings for children from birth through age five. The measures were developed by Westat and Child Trends with additional support from a group of experts who provided guidance and advice. The FPTRQ project created five measures of parent and provider/teacher relationships, including: 1) the director measure; 2) the provider/teacher measure; 3) the parent measure; 4) the family services staff measure; and 5) the family services staff parent measure. The FPTRQ measures integrate features from three perspectives of family-provider/teacher relationships in ECE: family support/ family-centered care; parent involvement/family involvement/family engagement, and family-sensitive caregiving (see Table 1-1). The measures incorporate the features that are consistent across these three perspectives as well as those that are unique to each of these perspectives. The Family and Provider/Teacher Relationship Quality Measures: Updated User's Manual1 introduces these measures, describes how they were developed, and provides in-depth information on how to use them. (author abstract)

Other

get fulltext

47.

Family and Provider/Teacher Relationship Quality measures: Updated user's manual [Executive summary]
Kim, Kwang, April, 2015
(OPRE Report 2014-65). Washington, DC: U.S. Administration for Children and Families, Office of Planning, Research and Evaluation.

The Family and Provider/Teacher Relationship Quality (FPTRQ) project, sponsored by the Administration for Children and Families' Office of Head Start (OHS) and Office of Planning, Research and Evaluation (OPRE), was a four-year effort to develop measures of the quality of family and provider/teacher relationships in early care and education (ECE) settings for children from birth through age five. The measures were developed by Westat and Child Trends with additional support from a group of experts who provided guidance and advice. The FPTRQ project created five measures of parent and provider/teacher relationships, including: 1) the director measure; 2) the provider/teacher measure; 3) the parent measure; 4) the family services staff measure; and 5) the family services staff parent measure. The FPTRQ measures integrate features from three perspectives of family-provider/teacher relationships in ECE: family support/ family-centered care; parent involvement/family involvement/family engagement, and family-sensitive caregiving (see Table 1-1). The measures incorporate the features that are consistent across these three perspectives as well as those that are unique to each of these perspectives. The Family and Provider/Teacher Relationship Quality Measures: Updated User's Manual1 introduces these measures, describes how they were developed, and provides in-depth information on how to use them. (author abstract)

Executive Summary

get fulltext

*
48.

Feasibility and acceptability of adapting the eating in the absence of hunger assessment for preschoolers in the classroom setting
Soltero, Erica G., December, 2015
Eating Behaviors, 19, 68-71

Eating in the Absence of Hunger (EAH) represents a failure to self-regulate intake leading to overconsumption. Existing research on EAH has come from the clinical setting, limiting our understanding of this behavior. The purpose of this study was to describe the adaptation of the clinical EAH paradigm for preschoolers to the classroom setting and evaluate the feasibility and acceptability of measuring EAH in the classroom. The adapted protocol was implemented in childcare centers in Houston, Texas (N = 4) and Phoenix, Arizona (N = 2). The protocol was feasible, economical, and time efficient, eliminating previously identified barriers to administering the EAH assessment such as limited resources and the time constraint of delivering the assessment to participants individually. Implementation challenges included difficulty in choosing palatable test snacks that were in compliance with childcare center food regulations and the limited control over the meal that was administered prior to the assessment. The adapted protocol will allow for broader use of the EAH assessment and encourage researchers to incorporate the assessment into longitudinal studies in order to further our understanding of the causes and emergence of EAH. (author abstract)

Reports & Papers

get fulltext

*
49.

Fidelity of implementation for an early-literacy intervention: Dimensionality and contribution to children's intervention outcomes
Guo, Ying, Q4 2016
Early Childhood Research Quarterly, 37(4), 165-174

This study examined fidelity of implementation (FOI) in the context of an early-literacy intervention involving 83 early childhood special education (ECSE) teachers and 291 three- to five-year old children with disabilities in their classrooms. Adherence, dosage, participant responsiveness, and program differentiation were assessed as multiple dimensions of FOI. Results demonstrated that a three-factor model of adherence and dosage, participant responsiveness, and program differentiation offered the best fit to the data to represent FOI. Further, program differentiation significantly related to children's early-literacy gains, and the effects of the intervention on children's gains in early literacy were fully mediated by program differentiation. Findings have implications for the design of effective early-literacy interventions and also for theorizing the construct of FOI. (author abstract)

Reports & Papers

*
50.

Finding and exploring existing large-scale data to study early care and education among Hispanics
Lopez, Michael, 13 October, 2016
New York: Child Care & Early Education Research Connections

This webinar, held on October 13, 2016, was hosted by Research Connections and the National Research Center on Hispanic Children and Families and was intended to help researchers learn about several new early care and education data resources. The National Research Center on Hispanic Children and Families recently released a series of data briefs and interactive tools to facilitate the access and use of existing, large-scale data sets to examine policy-relevant questions about early care and education use among low-income Hispanic families. The webinar introduced the series of four early care and education data briefs and described data sets that were included; described the use of the interactive tools; and provided two illustrative examples of how to use the briefs and interactive tools to access the most relevant data set(s) that will best help answer researchers' questions.

Multimedia

view multimedia

Select Citation

Search Feedback