Head Start: Further Development Could Allow Results of New Test to Be Used for Decision Making
GAO-05-343
Published: May 17, 2005. Publicly Released: May 17, 2005.
Skip to Highlights
Highlights
In September 2003, the Head Start Bureau, in the Department of Health and Human Services (HHS) Administration for Children and Families (ACF), implemented the National Reporting System (NRS), the first nationwide skills test of over 400,000 4- and 5-year-old children. The NRS is intended to provide information on how well Head Start grantees are helping children progress. Given the importance of the NRS, this report examines: what information the NRS is designed to provide; how the Head Start Bureau has responded to concerns raised by grantees and experts during the first year of implementation; and whether the NRS provides the Head Start Bureau with quality information.
Recommendations
Recommendations for Executive Action
Agency Affected | Recommendation | Status |
---|---|---|
Department of Health and Human Services | To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including monitoring the effects of the NRS on local Head Start instructional practices. |
As part of the National Reporting System (NRS) Quality Assurance effort, an independent contractor is collecting information from a national sample of 36 Head Start programs on how well they are implementing the NRS and what changes they seem to be making in curriculum and instructional practices in response to NRS requirements. Observed changes in instructional activities are documented in the periodic reports of the Quality Assurance contractor, which are carefully reviewed by Administration for Children and Families staff. Feedback on how programs can use the NRS to stimulate constructive changes has been and will be provided to local programs through the memoranda that accompany each report on NRS results and in nationwide satellite television broadcasts that focus on NRS data collection.
|
Department of Health and Human Services | To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including improving the management and accuracy of its data on the number of children eligible for and participating in the NRS. |
The Office of Head Start has requested that a National Reporting System contractor continue tracking major discrepancies between the number of children actually assessed and the number presumably eligible for assessment per the Program Information Report (PIR) and other sources of data.
|
Department of Health and Human Services | To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including working with the Technical Work Group to determine the feasibility of sampling options for administering the NRS, including documentation of their costs and benefits. |
The HHS contractor that developed the NRS prepared information regarding the costs and benefits of sampling options for review by HHS' Advisory Committee on Head Start Accountability and Educational Performance Measures. The Advisory Committee completed its work in February 2007, including a recommendation that HHS integrate the National Reporting System (NRS) into local assessment approaches. Congress has subsequently passed bills reauthorizing Head Start (H.R. 1429 and S. 556) that would suspend the NRS as currently implemented and ask the National Academy of Sciences to form a committee addressing developmental outcomes and assessments for young children.
|
Department of Health and Human Services | The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by determining how the NRS data will be used for the purposes of accountability and targeting training and technical assistance, and clearly communicate this information to grantees. |
In February 2007, the Secretary's Advisory Committee on Head Start Accountability and Performance Measures completed its work and recommended, in part, that HHS make clear to Head Start teachers the link between NRS results and technical assistance goals and that HHS determine appropriate gains and outcome levels so programs could meet their intended goals. HHS expressed the intention to consider these recommendations as it continued to improve the utility of the National Reporting System. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"
|
Department of Health and Human Services | The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by using the first year of NRS results to conduct further study to ensure that the results are reliable and valid for both the English and Spanish versions and that the results are appropriate for the intended purposes. |
An extensive series of analyses was conducted with the first year fall and spring data from the national implementation of the National Reporting System (NRS), as well as parallel data from the second year of national implementation. Data from the first year of implementation of the NRS suggest that the NRS measures have strong internal consistency reliability. Further, data from the original field test of the NRS suggest that these data also have strong inter-rater reliability and strong test-retest reliability. Taken together, these findings suggest that the NRS instruments are reliable. Similarly high reliability has been found for both the English and Spanish language versions of the NRS. These findings are currently being compiled in annual technical reports for the NRS that will be made available to the public upon completion. Further, a report on the predictive validity of the NRS, which is examined using FACES data, is currently being prepared. Preliminary results suggest that the NRS has strong predictive validity. Finally, a quality assurance study, which has been conducted every year since the implementation of the NRS, has shown high levels of fidelity in the implementation of the NRS direct child assessment. The findings of this report have been made available to grantees through the Computer-Based Reporting System.
|
Department of Health and Human Services | The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by compiling detailed technical information on the NRS, including appropriate uses, in a single, well-organized document and make this information publicly available. |
By the end of FY06, the contractor had prepared a technical report for the government's review that detailed both the process used to create the NRS, as well as findings from the NRS field test. As of August 2007, HHS officials indicated that the technical manual prepared by HHS' contractor was still in clearance. In 2008, HHS reported that the manual is publicly available upon request. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"
|
Full Report
Office of Public Affairs
Topics
Aid for educationData collectionData integrityEducation program evaluationEducational standardsEducational testingFederal aid programsPreschool educationReporting requirementsSurveys