Precollege Math and Science Education: Department of Energy's Precollege
Program Managed Ineffectively (Letter Report, 09/13/94, GAO/HEHS-94-208).
Public policymakers and U.S. industrial leaders have expressed grave
concern about precollege students in other industrialized countries
significantly outperforming American students on recent international
mathematics and science tests. In recognition of the Energy Department's
(DOE) world-class scientists and engineers as well as its
state-of-the-art research facilities, Congress made mathematics and
science education a major mission for DOE in fiscal year 1991. DOE's
precollege mathematics and science budget has grown about 1,250
percent--reaching about $27 million in fiscal year 1993. In the early
1990s, DOE did not effectively manage this program. First, DOE
jeopardized the program's success by not using a risk management
strategy to run the program's projects. Second, DOE forfeited an
invaluable management tool by taking a lax approach to program
evaluation. Third, DOE greatly reduced its chances of helping achieve
National Education Goal 5--"By the year 2000, U.S. students will be
first in the world in mathematics and science achievement"--by launching
a variety of projects that did not clearly seek to improve student
achievement. DOE has announced steps to substantially improve the
program's management and evaluation functions; however, the depth of
executive support for these measures and their subsequent staying power
are uncertain.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-94-208
TITLE: Precollege Math and Science Education: Department of
Energy's Precollege Program Managed Ineffectively
DATE: 09/13/94
SUBJECT: Secondary education
Agency missions
Energy research
Education program evaluation
Mathematics
Scientific research
Elementary education
Teacher education
Mission budgeting
Students
IDENTIFIER: National Education Goals
National Education Longitudinal Study of 1988
Goals 2000
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Chairman, Committee on Governmental Affairs, U.S.
Senate
September 1994
PRECOLLEGE MATH AND SCIENCE
EDUCATION - DEPARTMENT OF ENERGY'S
PRECOLLEGE PROGRAM MANAGED
INEFFECTIVELY
GAO/HEHS-94-208
Precollege Math and Science Education
Abbreviations
=============================================================== ABBREV
AMES - Ames Laboratory
ANL - Argonne National Laboratory
AWU - Associated Western Universities
BNL - Brookhaven National Laboratory
FCCSET - Federal Coordinating Council for Science, Engineering, and
Technology
CEHR - Committee on Education and Human Resources
LANL - Los Alamos National Laboratory
LBL - Lawrence Berkeley Laboratory
LLNL - Lawrence Livermore National Laboratory
NSF - National Science Foundation
ORISE - Oak Ridge Institute for Science and Education
ORNL - Oak Ridge National Laboratory
PNL - Pacific Northwest Laboratory
SNL-A - Sandia National Laboratory-Albuquerque
SNL-L - Sandia National Laboratory-Livermore
Letter
=============================================================== LETTER
B-251474
September 13, 1994
The Honorable John Glenn
Chairman, Committee on
Governmental Affairs
United States Senate
Dear Mr. Chairman:
Our nation's ability to remain the economic world leader depends on
its citizens' strong mathematics and science skills. Understandably,
public policymakers and industrial leaders have expressed grave
concern about precollege students in other industrialized countries
significantly outperforming American students on recent international
mathematics and science tests. In response to educational reform and
competency concerns, former President Bush and the nation's governors
developed six National Education Goals to be achieved by the year
2000.\1
In recognition of the Department of Energy's world-class scientists,
engineers, and technicians, as well as its state-of-the-art
laboratories and research facilities, the Congress made mathematics
and science education a major mission for the Department in fiscal
year 1991. Consequently, Energy's precollege mathematics and science
program budget has grown approximately 1,250 percent--from
approximately $2 million in fiscal year 1990 to approximately $27
million in fiscal year 1993 (see app. I).
This report responds to your questions about how effectively Energy
manages its precollege mathematics and science program. On the basis
of discussions with your office, we agreed to determine (1) the
appropriateness of Energy's precollege program implementation
priorities, (2) the role of project evaluations in ensuring rational
budget decisions, and (3) whether its precollege program helps
achieve National Education Goal 5--"By the year 2000, U.S. students
will be first in the world in mathematics and science achievement."
--------------------
\1 In 1994, the Goals 2000: Educate America Act expanded the
National Education Goals from six to eight. The goals address (1)
school readiness; (2) school completion; (3) student achievement and
citizenship; (4) teacher education and professional development; (5)
mathematics and science achievement;(6) adult literacy; (7) safe,
disciplined, and alcohol- and drug-free schools; and (8) parental
participation. The National Education Goals were originally
developed in 1989.
BACKGROUND
------------------------------------------------------------ Letter :1
Over the last 10 years, numerous reports have charged that many U.S.
students complete high school scientifically and technologically
illiterate. According to these reports, not only are U.S. students
less well educated than their predecessors, they are also less well
trained in mathematics and science than their peers in other
industrialized countries. Reported decreasing student enrollments in
science courses, declining achievement test scores, and the
continuing decline in the number of high-quality mathematics and
science teachers have highlighted problems in precollege mathematics
and science instruction.\2
To remedy this perceived crisis in education, the Congress conferred
mathematics and science education responsibilities on the Department
of Energy and 13 other federal agencies--in addition to the
Department of Education and the National Science Foundation (NSF).
To improve mathematics and science education, the Federal
Coordinating Council for Science, Engineering, and Technology
(FCCSET)\3 Committee on Education and Human Resources (CEHR) was
charged with coordinating the efforts of these 16 agencies. FCCSET
developed broad implementation priorities for precollege education,
including (1) standards for curriculum, teaching, and assessment; (2)
curriculum, course, and instructional materials; (3) systemic reform;
and (4) teacher preparation and enhancement. These priorities were
formalized in January 1993 as part of a 5-year strategic plan.
--------------------
\2 Science, Engineering, and Mathematics Education, Congressional
Research Service Issue Brief, (Washington, D.C.: 1992), p. 1.
\3 In November 1993, President Clinton established by executive order
a cabinet level National Science and Technology Council (NSTC) to
coordinate science, space, and technology policies throughout the
federal government. The establishment of NSTC consolidated the
responsibilities previously carried out by a number of agencies,
including FCCSET.
ENERGY'S PRECOLLEGE
MATHEMATICS AND SCIENCE
PROGRAM
---------------------------------------------------------- Letter :1.1
The Department of Energy Science Education Enhancement Act authorized
Energy to undertake a wide range of precollege education and training
activities. Some of Energy's activities include making loans of
equipment and staff to schools; allowing Energy employees to provide
education-oriented community services; and participating in joint
programs with schools, businesses, museums, and other community
partners. Energy manages its precollege program with a decentralized
organizational structure: Energy's nine national laboratories and 22
of its research facilities were given the flexibility to design and
implement projects using the broad implementation priorities provided
by FCCSET. In designing projects, each Energy facility considered
FCCSET's broad implementation priorities, as well as its individual
areas of specialization and local needs. Although the projects vary,
most emphasize hands-on experiences and fall within three
implementation categories:
Teacher enhancement--These projects attempt to further the content
knowledge, skills, and experiences of teachers already in the
workforce. Teacher enhancement projects typically provide
teachers with opportunities to (1) work on a variety of
scientific and technical subjects as a member of an Energy
laboratory research team, (2) train with mentors who assist them
with in-school science experiments, or (3) obtain sophisticated
computer training.
Student support--These projects seek to reward students for
outstanding achievement, afford them enrichment experiences, and
furnish them with supplementary educational services such as
tutoring and mentoring. Student support projects typically
allow students opportunities to (1) participate in cutting-edge
research at an Energy laboratory, (2) attend scientific lectures
and demonstrations, and (3) study emerging topics such as
environmental energy.
Systemic reform--These projects aim to improve education by
changing all aspects of an educational system. Systemic reform
projects typically involve key education stakeholders--students,
teachers, administrators, policymakers, and parents--in
collaborative efforts to (1) create goals and standards for all
students, (2) develop related curricula and instructional
materials, and (3) provide professional development for
teachers.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Although Energy invested more than $50 million in precollege
education in fiscal years 1990-93, it did not effectively oversee or
direct the program. For example, although research findings indicate
that systemic reform may have the greatest potential to improve
student learning, projects in this implementation category
constituted the smallest share of Energy's precollege budget, about
11 percent. In contrast, Energy used about 70 percent of its
precollege budget to finance teacher enhancement projects, even
though research suggests that these projects may be ineffective at
increasing student achievement.
To compound problems, Energy did not link budget decisions to project
evaluation results. As a result, Energy had not evaluated almost
half of its 17 most resource-intensive projects at the time of our
review; for those projects with evaluation reports, all were
inadequate. Nonetheless, Energy substantially increased funding for
most of these projects--in one case by over 1,700 percent.
In addition, it is doubtful whether Energy's precollege program will
help achieve National Education Goal 5. In this regard, Energy's
projects typically do not focus on student achievement, which is
central to achieving this goal. In fact, more than half of Energy's
most resource-intensive projects did not directly include improving
student achievement as an objective.
During our review, Energy indicated recognition of the need to pay
closer attention to managing its precollege program. To correct this
situation, Energy recently drafted an agency-specific strategic plan.
In addition, officials said they plan to restructure their program to
ensure that all projects are evaluated and linked more clearly to
National Education Goal 5.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
Our review of Energy's precollege mathematics and science program
focused primarily on its 288 fiscal year 1992 projects. To
accomplish our objectives, we conducted site visits at Energy's
headquarters in Washington, D.C., and eight laboratories and research
facilities in California, Illinois, New Mexico, Tennessee, and Utah,
which administered the most resource-intensive precollege projects.\4
During these site visits, we interviewed laboratory personnel as well
as school administrators, teachers, and students involved in the
precollege program.
To obtain an overview of the program, we collected general
information on Energy's entire array of fiscal year 1992 precollege
mathematics and science projects and reviewed the literature on the
relationship between teacher quality and subsequent student
achievement. For the 17 most resource-intensive projects, we
collected and analyzed budget data for fiscal years 1990-93 (see app.
III) and, when available, evaluation reports. Although constituting
less than 6 percent of Energy's program portfolio, these 17 projects
accounted for 50 percent ($11 million) of total program dollars in
fiscal year 1992. To determine whether the evaluation reports
provided reliable information on project effectiveness, we also
conducted a technical review of the nine project evaluation reports
Energy officials submitted to our staff (see app. IV).
We conducted our work between November 1992 and July 1994 in
accordance with generally accepted government auditing standards.
--------------------
\4 To identify the most resource-intensive projects, we examined the
extent to which both financial and personnel resources were used. We
selected 16 projects with the largest budgets and 1 project that
involved considerably more Energy personnel than other projects.
FOCUS ON TEACHER ENHANCEMENT
PROJECTS IS QUESTIONABLE
------------------------------------------------------------ Letter :4
Energy's decision to concentrate its precollege program resources on
teacher enhancement projects, which account for more than two-thirds
of the program budget, is a questionable implementation strategy. In
an earlier report, we found no evidence that training programs to
upgrade existing science and mathematics teachers' skills will
improve teaching effectiveness.\5 In reviewing more recent studies,
we found mixed results: in some instances, researchers found small
yet statistically significant positive correlations between teacher
knowledge and student achievement; in others, researchers failed to
demonstrate any significant correlations. Conversely, current
literature suggests that systemic reform measures, such as
high-quality curriculum development, may hold the most promise for
improving academic achievement and realizing the national math and
science goals.
--------------------
\5 New Directions for Federal Programs to Aid Mathematics and Science
Teaching (GAO/PEMD-84-5, Mar. 6, 1984).
PRECOLLEGE PROGRAM HEAVILY
WEIGHTED TOWARD TEACHER
ENHANCEMENT PROJECTS
---------------------------------------------------------- Letter :4.1
Both in budget dollars and project numbers, Energy devoted most of
its precollege program resources to teacher enhancement projects
during fiscal year 1992. Regarding the program budget, Energy spent
about 70 percent ($15.4 million) of all precollege program dollars to
upgrade teachers' mathematics and science skills--45 percent
exclusively for teacher enhancement and another 25 percent for
teacher enhancement combined with a student support component. For
the residual, about 19 percent ($4.2 million) of Energy's precollege
budget focused exclusively on student support; systemic reform
efforts constituted just 11 percent ($2.4 million). Regarding the
total number of projects, 157 of 288 (55 percent) had a substantial
teacher enhancement component, another 113 projects (39 percent)
focused exclusively on student support, 14 projects (5 percent)
involved systemic reform efforts, and 4 projects (1 percent) included
activities that did not specifically involve teachers and students
(see table 1).
Table 1
Precollege Program Heavily Weighted
Toward Teacher Enhancement Projects
Budget
(in
thousa Percen Percen
Implementation priorities nds) t Number t
---------------------------- ------ ------ ------ ------
Teacher enhancement $9,911 45 100 35
Teacher enhancement and 5,524 25 57 20
student support
Student support 4,187 19 113 39
Systemic reform 2,448 11 14 5
Other\a 95 \b 4 1
============================================================
Total $22,16 100 288 100
5
------------------------------------------------------------
\a "Other" includes activities that do not specifically involve
students and teachers, such as efforts by Energy's staff to develop a
catalog listing precollege physics projects.
\b Less than 1 percent.
NO STRONG RELATIONSHIP
BETWEEN TEACHER ENHANCEMENT
AND STUDENT ACHIEVEMENT
---------------------------------------------------------- Letter :4.2
Research has failed to show conclusively a relationship between
teacher enhancement and student achievement.\6 We based our finding
on the results of a 1984 GAO report and a review of recent studies
that examined the effect of teacher quality on student achievement
(see bibliography, p. 25). In our earlier report, we found no
evidence that training programs to upgrade mathematics and science
teachers' skills improved student achievement.\7 That analysis was
based first on an NSF study that compared the achievement of eighth
and eleventh grade students whose teachers participated in NSF
institutes to those who did not. The study showed that teacher
participation in NSF institutes had a positive effect on eleventh
grade students' science and mathematics achievement. However,
institute participation did not have a statistically significant
effect on eighth grade student achievement in either science or
mathematics. Second, several general studies from the 1960s and
1970s as a group failed to show a consistent relationship between
teacher knowledge and student achievement.
More recent studies have also failed to demonstrate a strong
relationship between teacher enhancement/knowledge and student
achievement. Two of the most prominent studies conducted between
1984 and 1994 that examined this relationship reported mixed results:
A 1992 NSF study, which analyzed teacher transcript and student
test data from the National Education Longitudinal Study of 1988
(NELS:88),\8 showed a statistically significant relationship
between eighth grade students' mathematics achievement and their
teachers' preparation.\9 Specifically, students whose teachers
had majored in mathematics performed slightly better than those
whose teachers had majored in education only. However, no
statistically significant relationship existed for science.
A 1994 Chicago Academy of Sciences study, which analyzed the effect
of teachers' participation in 4-week summer science workshops,
reported a small, but statistically significant, increase in the
level of science achievement for seventh grade students of
participants.\10 For these students, the average science
achievement score increased from 47.1 to 49.6. However, no
statistically significant change occurred for either sixth or
eighth grade science students.
At least four explanations for the weak relationship between
teachers' knowledge or participation in training programs and student
achievement exist. First, most training programs generally involve
one-time, relatively short events with little or no follow-up. For
example, a single 4- to 8-week summer research experience probably
would not produce a dramatic change in a teacher's effectiveness.
Second, teacher training programs are subject to self-selection bias;
that is, in-service training often attracts exemplary or highly
motivated teachers. Exposing such teachers to short-term workshop
training may not significantly add to their teaching effectiveness.
Third, the most knowledgeable teachers may not be the best teachers.
Other factors besides teacher knowledge--such as enthusiasm,
confidence, and organization of class time--may determine student
achievement. Fourth, some researchers suggest that current student
assessments inadequately measure the higher order, problem-solving
skills, which could be affected by teacher training and knowledge.
--------------------
\6 Although research in this area has been limited, the studies we
identified continue to be cited in current research, and the findings
remain unchallenged.
\7 New Directions for Federal Programs to Aid Mathematics and Science
Teaching.
\8 NELS:88 is a nationally representative sample of 26,435
eighth-grade students clustered within 1,052 schools.
\9 Senta Raizen and Theodore Britton, "Science and Mathematics
Teachers," Indicators of Science and Mathematics Education in 1992,
National Science Foundation (Washington, D.C.: 1993), pp. 85-113.
\10 Jon D. Miller, Enriching Middle School Science: A Final
Evaluation of the 1991-92 Columbia College Workshops Utilizing an
Innovative Approach to the Teaching of Science. Chicago Academy of
Sciences (NSF grant TPE 89-55128), (Chicago: 1994), pp. 30-31.
SYSTEMIC REFORM CONSIDERED
PROMISING APPROACH FOR
INCREASING STUDENT
ACHIEVEMENT
---------------------------------------------------------- Letter :4.3
Energy has implemented few systemic reform projects, even though
current educational literature suggests systemic reform may have the
greatest potential for improving student learning.\11 Systemic reform
is promising because it (1) attempts to stimulate change in many or
all components of the educational system simultaneously; (2)
establishes clear standards for what students should know and be able
to do; and (3) involves key educational stakeholders--students,
teachers, administrators, teacher educators, textbook publishers,
policymakers, and parents--at all levels of the education
system--national, state, district, and school.
Under systemic reform, teacher training ideally takes place in an
environment that supports new curricula or teaching techniques
learned during training. However, many educators believe that
retraining individual teachers will have little measurable impact on
student achievement if the education system is not prepared to absorb
improvements. For example, although teacher enhancement programs may
provide teachers the knowledge, skills, and enthusiasm essential to
implement new curricula, teachers can rarely implement and sustain a
new program if their school support systems and attitudes of
administrators, colleagues, and parents have not changed.
--------------------
\11 Much of this literature is cited in Marshall Smith and Jennifer
O'Day, "Systemic School Reform," Politics of Education Association
Yearbook, (1990), pp. 233-267.
ADDITIONAL PROGRAM DIVERSITY
COULD REDUCE PROGRAM RISK
---------------------------------------------------------- Letter :4.4
Given the evidence cited, Energy's heavy investment in teacher
enhancement projects substantially increases its risk of not
improving student achievement in mathematics and science. However,
Energy could reduce this risk by changing its mix of projects to
balance the program, much like financial advisers do by diversifying
investment portfolios. In managing uncertainty, financial advisers
minimize the risk of loss by acquiring a variety of investment
vehicles; thus, good returns from one investment counterbalance poor
returns from another. Building on this analogy, Energy could view
its precollege program as a collection of investment vehicles
assembled to meet an investment goal--improved student achievement in
mathematics and science. Thus, given the uncertainty of its
projects' educational payoffs, a more diverse program portfolio would
enhance the likelihood of Energy's achieving its program goal.
To address these concerns, Energy developed a strategic plan, which
identifies agency-specific precollege goals and objectives in March
1994. Energy envisions using the strategic plan, which should be in
place by November 1994, to help it create a program strategy that
supports the best mix of projects and minimizes risk. In addition,
Energy plans to look at the feasibility of eliminating projects that
do not support its strategic plan and restructuring all teacher
enhancement projects to include systemic reform elements, such as
follow-up support beginning in fiscal year 1995.
BUDGET DECISIONS NOT LINKED TO
PROJECT EVALUATIONS
------------------------------------------------------------ Letter :5
Energy did not link budget decisions to project evaluations. Until
recently, Energy neither required project evaluations nor ensured
their adequacy when its research facilities conducted them. For
example, Energy elected not to evaluate eight of its 17 most
resource-intensive projects. When conducted, evaluations were of
poor quality. For instance, all projects with evaluations contained
discrediting technical flaws--such as insufficient sample sizes, the
absence of statistical tests, and insufficient supporting data--that
potentially invalidated any evaluation findings (see app. IV).
Energy's limited use of program evaluation reflected its management
priorities. According to Energy officials, the Department did not
emphasize effectiveness evaluations because program expansion was its
primary objective. These officials also said that Energy lacked the
capacity--staff, funds, and expertise--to design and monitor
evaluations for such a vast array of projects. Consequently, in the
absence of sufficient evaluation results, Energy substantially
increased project budgets on the basis of self-reported data, such as
customer satisfaction surveys, or anecdotal data such as participant
testimonials, requests to participate in particular projects, or
other popularity indicators. Consider the following examples:
Energy increased the budgets for four projects with no evaluations,
with increases ranging from 62 to 912 percent from fiscal year
1991 to 1992 (see fig. 1).\12
Figure 1: Project Budgets
Increased Substantially Without
Evaluations
(See figure in printed
edition.)
Notes:
\a The Science/Math Carnival (SNL-L) project budget was increased
from approximately $3,000 to $26,000 or by 912 percent.
\b The Bay Area Science and Technology Education Collaboration (LBL)
project budget was increased from approximately $288,000 to $466,000
or by 62 percent.
\c The Environmental Management Precollege Analytical Chemistry (AWU)
project budget was increased from approximately $225,000 to $615,000
or by 173 percent.
\d The National Science Explorers (ANL) project budget was increased
from approximately $318,000 to $693,000 or by 118 percent.
Energy increased the budgets for five projects with inadequate
evaluations, with increases ranging from 113 to 1,731 percent
from fiscal year 1990 to 1992 (see fig. 2).
Figure 2: Project Budgets
Increased Substantially Despite
Inadequate Evaluations
(See figure in printed
edition.)
Notes:
\a The Environmental Education Outreach for Minorities (BNL) project
budget was increased from approximately $93,000 to $199,000 or by 113
percent.
\b The Teacher Research Associates (ORISE) project budget was
increased from approximately $49,000 to $231,000 or by 372 percent.
\c The OPTIONS (PNL) project budget was increased from approximately
$50,000 to $915,000 or by 1,731 percent.
\d The Teacher Research Associates (AWU) project budget was increased
from approximately $526,000 to $1,779,000 or by 238 percent.
\e The Science Advisors (SNL-A) project budget was increased from
approximately $545,000 to $2,500,000 or by 359 percent.
On the basis of our findings, Energy needs to change the way it views
the relationship between project implementation and evaluation.
Energy should view program evaluation as an integral part of program
management; rather than perceiving the two as mutually exclusive.
Because quality program evaluation is expensive, Energy must accept
the unavoidable trade-off that it must fund fewer projects with
stronger evaluation components.
To address its program evaluation shortcomings, Energy began a
partnership with the National Center for Improving Science Education
to jointly develop a system for ongoing evaluation of its precollege
program in May 1992. In addition, Energy established an evaluation
guidance committee responsible for developing a long-range
implementation plan for project evaluation. Energy officials also
said that beginning in fiscal year 1995, the Department will require
each precollege project to have an evaluation component as a
prerequisite for funding.
--------------------
\12 Projects shown in figure 1 began in fiscal year 1991.
PROGRAM UNLIKELY TO CONTRIBUTE
TO ACHIEVING NATIONAL EDUCATION
GOAL 5
------------------------------------------------------------ Letter :6
Energy has greatly diminished the prospect of its program's helping
to achieve National Education Goal 5--making American students first
in the world in mathematics and science--by not emphasizing student
achievement. Although it is the essence of Goal 5, increasing
student achievement is the key objective in only 7 of the 17 most
resource-intensive program projects. The remaining 10 projects
generally seek to improve students' attitudes toward mathematics and
science and motivate them to eventually pursue science careers by
improving their perceptions of scientists.
Moreover, Energy's evaluation process did not focus on student
achievement: only one of the nine projects that were evaluated tried
to measure student achievement. In fact, Energy's projects are
seldom clearly linked to National Education Goal 5. For example, two
of seven program managers we interviewed were unfamiliar with the
National Education Goals. In addition, none of the program managers
could demonstrate how their projects helped improve student
achievement. Generally, these managers told us their projects aim to
increase mathematics and science literacy and promote science as a
career, not improve student achievement.
CONCLUSION
------------------------------------------------------------ Letter :7
In the early 1990s, Energy did not effectively manage its precollege
mathematics and science program. First, the Department jeopardized
the program's success by not using a risk management strategy to
administer the program's projects. Second, Energy forfeited an
invaluable management tool by taking a lax approach to program
evaluation. Third, Energy greatly reduced its probability of helping
achieve National Education Goal 5 by implementing a variety of
projects that did not clearly seek to improve student achievement.
In response to concerns raised during our review, Energy officials
announced plans to undertake several initiatives to substantially
improve this program's management and evaluation functions. These
initiatives constitute an important step toward effective program
management. However, the depth of executive support for these
initiatives and their subsequent staying power were uncertain at the
time our review was completed. If ongoing changes in Energy's
management philosophy are fully implemented, needed program
improvements could ultimately result.
RECOMMENDATIONS TO THE
SECRETARY OF ENERGY
------------------------------------------------------------ Letter :8
In continuing the Department's efforts to improve management of the
precollege mathematics and science education program, we recommend
that the Secretary of Energy strengthen its management role.
Specifically, the Secretary should
place greater emphasis on balancing the program by increasing the
proportion of systemic reform projects;
strengthen its evaluation component so that it serves as a basis
for (1) improving projects; (2) making informed budget decisions
about terminating, retaining, and expanding projects; and (3)
measuring gains in student achievement; and
restructure or discontinue all projects that do not clearly support
National Education Goal 5--increasing students' mathematics and
science achievement.
---------------------------------------------------------- Letter :8.1
Department of Energy officials who reviewed a draft of this report
generally agreed with our findings. As requested, unless you
publicly announce its contents earlier, we plan no further
distribution of this report until 30 days from its issue date. At
that time, we will send copies of this report to the appropriate
House and Senate Committees, the Secretary of Energy, and other
interested parties. If you or your staff have any questions about
this report, please call me on (202) 512-7014 or Cornelia Blanchette,
Associate Director, on (202) 512-8403. The major contributors to
this report are listed in appendix V.
Sincerely yours,
Linda G. Morra
Director, Education and
Employment Issues
PRECOLLEGE PROGRAM BUDGET
INCREASED SIGNIFICANTLY (FISCAL
YEARS 1990-93)
=========================================================== Appendix I
(See figure in printed
edition.)
Note: Actual budget totals shown for fiscal years 1990-92.
Appropriated budget totals shown for fiscal year 1993.
OVERVIEW OF 17 MOST
RESOURCE-INTENSIVE PROJECTS
(FISCAL YEAR 1992)
========================================================== Appendix II
Fiscal
year
Energy 1992 Implementation
facility Project name Project objective budget priority
-------------- ----------------- ----------------- ---------- --------------
Projects with no evaluation (eight projects)
--------------------------------------------------------------------------------
Ames Adventures in Introduce $315,888 Teacher
Laboratory Supercomputing teachers to the enhancement
(Iowa) use of high-
performance
computers in
mathematics and
science
instruction.
Argonne National Science Provide teachers 693,000 Teacher
National Explorers with science enhancement
Laboratory videos,
(Illinois) instructional
guides, and
training
workshops to
improve science
teaching.
Associated Environmental Provide students 615,014 Teacher
Western Management with an enhancement
Universities Precollege opportunity to and student
(Utah) Analytical take a college- support
Chemistry level analytical
chemistry course.
Lawrence Bay Area Science Provide hands-on 466,000 Systemic
Berkeley & Technology activities, reform
Laboratory Education curriculum
(California) Collaboration development,
instructional
materials, and
training
districtwide to
enhance classroom
teaching and
learning of
science,
mathematics, and
technology.
Oak Ridge Science & Provide teachers 244,000 Systemic
Institute for Mathematics and students with reform
Science and Action for research
Education Revitalized opportunities,
(Tennessee) Teaching instructional
materials, and
technical support
to increase the
effectiveness of
mathematics and
science education
districtwide.
Oak Ridge Adventures in Provide teachers 398,000 Teacher
National Supercomputing and students with enhancement
Laboratory access to and and student
(Tennessee) training on high- support
performance
computers to
improve
mathematics and
science
instruction.
Oak Ridge Preparation & Provide teachers 260,000 Teacher
National Enhancement with summer enhancement
Laboratory research
(Tennessee) opportunities and
training to
improve
mathematics and
science
instruction.
Sandia Science/Math Provide 26,306 Teacher
National Carnival scientific enhancement
Laboratory- demonstrations to
Livermore improve
(California) mathematics and
science teaching.
Subtotal $3,018,208
Projects with evaluation (nine projects)
--------------------------------------------------------------------------------
Argonne Chicago Science Provide teachers $531,000 Teacher
National Explorers and students with enhancement
Laboratory hands-on and student
(Illinois) activities, field support
trips, and videos
to improve
science
instruction and
learning.
Associated Teacher Research Provide summer 1,779,269 Teacher
Western Associates research enhancement
Universities opportunities to
(Utah) teachers.
Brookhaven Environmental Provide students 199,211 Student
National Education with an support
Laboratory Outreach for opportunity to
(New York) Minorities take a college-
level
environmental
science course.
Brookhaven Northeast Provide students 229,381 Student
National Consortium for with an support
Laboratory Minorities opportunity to
(New York) take college-
level science
courses.
Los Alamos Students Watching Provide 1,016,900 Teacher
National Over Our Planet materials, enhancement
Laboratory Earth teacher training, and student
(New Mexico) and student support
instruction on
environmental
concerns.
Lawrence National Provide access to 726,000 Teacher
Livermore Education and training on enhancement
National Supercomputer high-performance and student
Laboratory Program computers to support
(California) students and
teachers.
Oak Ridge Teacher Research Provide summer 230,600 Teacher
Institute for Associates research enhancement
Science and opportunities for
Education teachers.
(Tennessee)
Pacific OPTIONS in Provide students 915,370 Systemic
Northwest Science with high- reform
Laboratory quality
(Washington) mathematics and
science education
by enhancing
teachers'
instructional
strategies and
ability to
develop
curriculum
through a
statewide
systemic reform
effort.
Sandia Science Advisors Provide a 2,500,000 Teacher
National scientist in the enhancement
Laboratory- school who offers and student
Albuquerque technical support
(New Mexico) assistance to
teachers and
students and
participates in
activities to
support science
(i.e., science
fairs).
Subtotal $8,127,731
Total $11,145,93
9
--------------------------------------------------------------------------------
PERCENT CHANGE IN BUDGET FOR
ENERGY'S
17 MOST RESOURCE-INTENSIVE
PROJECTS
(FISCAL YEARS 1990-93)
========================================================= Appendix III
Per
cen Perc
t ent Percen
Bud cha chan t
Energy facility Project name get Budget nge Budget ge Budget change
------------------ ---------------- --- ---------- --- ------------ ---- ------------ ------
Projects with no evaluation (eight projects)
----------------------------------------------------------------------------------------------------
Ames Laboratory Adventures in $315,888 N/A $650,000 106
(Iowa) Supercomputing
Argonne National National Science $318,000 N/ 693,000 118 507,000 -27
Laboratory Explorers A
(Illinois)
Associated Western Environmental 225,117 N/ 615,014 173 601,870 -2
Universities Management A
(Utah) Precollege
Analytical
Chemistry
Lawrence Berkeley Bay Area Science 288,300 N/ 466,000 62 515,000 11
Laboratory & Technology A
(California) Education
Collaboration
Oak Ridge Science & 24, 279,100 1,0 244,000 -13 260,000 7
Institute for Mathematics 100 58
Science and Action for
Education Revitalized
(Tennessee) Teaching
Oak Ridge National Adventures in 398,000 N/A 920,000 131
Laboratory Supercomputing
(Tennessee)
Oak Ridge National Teacher 260,000 N/A 260,000 N/A
Laboratory Preparation &
(Tennessee) Enhancement
Sandia National Science/Math 2,600 N/ 26,306 912 29,000 10
Laboratory - Carnival A
Livermore
(California)
Subtotal $24 $1,113,117 4,5 $3,018,208 171 $3,742,870 24
,10 19
0
Projects with evaluation (nine projects)
----------------------------------------------------------------------------------------------------
Argonne National Chicago Science $61 $570,000 -7 $531,000 -7 $750,000 41
Laboratory Explorers 3,0
(Chicago) 00
Associated Western Teacher Research 525 1,105,519 110 1,779,269 61 1,731,326 -3
Universities Associates ,79
(Utah) 1
Brookhaven Environmental 93, 125,358 34 199,211 59 168,500 -15
National Education 351
Laboratory (New Outreach for
York) Minorities
Brookhaven Northeast 122,503 N/ 229,381 87 249,700 9
National Consortium for A
Laboratory (New Minorities
York)
Los Alamos Students 85, 750,000 782 1,016,900 36 289,000 -72
National Watching Over 000
Laboratory (New Our Planet
Mexico) Earth
Lawrence Livermore National 340,000 N/ 726,000 114 403,000 -45
National Education A
Laboratory Supercomputer
(California)
Oak Ridge Teacher Research 48, 139,700 186 230,600 65 193,000 -16
Institute for Associates 900
Science and
Education
(Tennessee)
Pacific Northwest OPTIONS in 50, 629,762 1,1 915,370 45 374,033 -59
Laboratory Science 000 60
(Washington)
Sandia National Science Advisors 545 1,400,000 157 2,500,000 79 2,200,000 -12
Laboratory - ,00
Albuquerque (New 0
Mexico)
====================================================================================================
Subtotal $1, $5,182,842 164 $8,127,731 57 6,358,559 -22
961
,04
2
====================================================================================================
Total $1, $6,295,959 217 $11,145,939 77 $10,101,429 -9
985
,14
2
----------------------------------------------------------------------------------------------------
Note: N/A represents not applicable.
EVALUATION REPORT RESULTS FOR NINE
PROJECTS WITH EVALUATIONS
========================================================== Appendix IV
(See figure in printed
edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
Wayne B. Upshaw, Assistant Director
Valerie Giles-Reynolds, Assignment Manager, (313) 256-8000
Lemuel Jackson, Evaluator-in-Charge
Ella Cleveland
Joel Grossman
Revae E. Steinman
Yelena K. Thompson
BIBLIOGRAPHY
=========================================================== Appendix 0
Begle, Edward. Teacher Knowledge and Student Achievement in Algebra.
School Mathematics Study Group. (Palo Alto, Calif.:1972).
Darling-Hammond, Linda. "Enlarging the Knowledge Base--Are Our
Teachers Ready To Teach?" National Council for Accreditation of
Teacher Education, 1991.
Darling-Hammond, Linda, and Lisa Hudson. Indicators for Monitoring
Mathematics and Science Education: A Sourcebook. ed. Shavelson,
McDonnell, and Oakes. RAND and the National Science Foundation,
1989.
Druva, Cynthia A., and Ronald D. Anderson, "Science Teacher
Characteristics by Teacher Behavior and by Student Outcome: A
Meta-Analysis of Research." Journal of Research in Science Teaching,
Vol. 20, No. 5 (1983), pp. 467-479.
Fitzsimmons, Stephen J. A Study of NSF Teacher Enhancement Program
(TEP) Participants and Principal Investigators: 1984-1989 (DRAFT)
Vol. I: Summary Report. Abt Associates, Inc. for the National
Science Foundation. Cambridge, Mass.: 1993.
Juster, Thomas, F. Precollege Science and Mathematics Teachers:
Monitoring Supply, Demand and Quality. National Research Council,
Committee on National Statistics, Commission on Behavioral and Social
Sciences and Education, 1990.
Lawrenz, Frances. "The Relationship Between Science Teacher
Characteristics and Student Achievement and Attitude," Journal of
Research in Science Teaching, Vol. 12, No. 4 (1975), pp. 433-37.
McKinney, Kay. Improving Math and Science Teaching. U.S.
Department of Education, Office of Educational Research and
Improvement. Washington, D.C.: 1992.
Miller, Jon. Enriching Middle School Science: A Final Evaluation of
the 1991-92 Columbia College Workshops Utilizing an Innovative
Approach to the Teaching of Science. Chicago Academy of Sciences.
Chicago: 1994.
Murnane, Richard J., and Senta A. Raizen, Improving Indicators of
the Quality of Science and Mathematics Education in Grades K-12.
National Research Council, Committee on Indicators of Precollege
Science and Mathematics Education, Commission on Behavioral and
Social Sciences and Education, 1988.
Raizen, Senta, and Theodore Britton, "Science and Mathematics
Teachers," Indicators of Science and Mathematics Education--1992.
National Science Foundation. (NSF 93-95). Washington, D.C.: 1993.
Rothman, Arthur I. "Teacher Characteristics and Student Learning,"
Journal of Research in Science Teaching, Vol. 6 (1969), pp. 340-48.
Smith, Marshall, S., and Jennifer O'Day, "Systemic School Reform."
Politics of Education Association Yearbook, 1990, pp. 233-267.
U.S. General Accounting Office. New Directions for Federal Programs
to Aid Mathematics and Science Teaching, GAO/PEMD-84-5. Washington,
D.C.: 1984.
U.S. General Accounting Office. Systemwide Education Reform:
Federal Leadership Could Facilitate District-Level Efforts,
GAO/HRD-93-97. Washington, D.C.: 1993.