ERIC
ERIC Identifier: ED458215
Publication Date: 2001-05-00
Author: Tienken, Christopher - Wilson, Michael
Source: ERIC Clearinghouse on Assessment and Evaluation College Park MD.

State Standards, Assessments and Instruction. ERIC Digest.


THIS DIGEST WAS CREATED BY ERIC, THE EDUCATIONAL RESOURCES INFORMATION CENTER. FOR MORE INFORMATION ABOUT ERIC, CONTACT ACCESS ERIC 1-800-LET-ERIC

Most states have put in place curriculum standards and state-developed assessments to monitor the implementation of those standards. Most state standards define expected outcomes, that is, what students need to know and be able to do, but do not mandate specific strategies or pedagogy used by local districts. As researchers Paul Black and Dylan William (1998) note, standards are raised only by changing what happens in the classroom, beginning with teachers and students. This Digest describes a program used by two educators to help teachers improve instruction through a deeper understanding of state standards and test specifications.

DEVELOPING A KNOWLEDGE BASE

Understanding how standards-based state tests are constructed is the first step in being able to use them to guide and improve instruction. A test is essentially a sample of questions or activities that reflect a large body of knowledge and mental processes associated with an academic subject area such as language arts, mathematics, science, etc. School districts are under pressure to perform well on state tests and often use a test preparation strategy of giving students sample tests from commercially prepared workbooks or state-released items to get ready for state tests. Although this strategy can be useful for providing general information regarding student strengths and weaknesses as related to the samples, it should not be the only method used by teachers because it does little to educate them about how to understand and use state tests, standards, and test specifications. This article recommends a three-part process of delineation, alignment, and calibration for developing an understanding of state assessments and using that understanding to improve instruction. The New Jersey Core Curriculum Content Standards are used to model the process, but it can be applied to any state's or district's standards.

DELINEATION

Delineation is the process of identifying all aspects or dimensions of a particular subject domain. It involves using state testing documents that describe each content area of the assessment, including test specifications, specific skill cluster information, subject area frameworks, assessment examples and exemplars, and the state standards. Delineation requires an examination of these documents for assessment dimensions such as content, cognitive level and complexity. A thorough delineation might also include analysis of additional factors such as the test format and difficulty level of the questions.

The New Jersey Standards include macro, or big picture, statements and cumulative progress indicators that provide details about general performance expectations. The standards include knowledge specifications, which describe the specific processes and content that all students must know by the end of fourth grade (also known as content standards) as well as problem-solving specifications, which describe what students should be able to do with the content knowledge (also known as process standards). The following example is excerpted from the 4th grade New Jersey mathematics standards and test specification manuals.

Macro Standard 4.1: All students will develop the ability to pose and solve mathematical problems in mathematics, other disciplines, and everyday experiences.

Cumulative Progress Indicator 4.1.2: Recognize, formulate, and solve problems arising from mathematical situations and everyday experiences. Test Specification Manual - Cluster IV Discrete Mathematics: Knowledge (content standards): Students should have a conceptual understanding of: Tree diagram

Problem Solving (process standards): In problem solving settings, students should be able to: Draw and interpret networks and tree diagrams

After reviewing the entire body of standards and test specifications for fourth-grade mathematics in New Jersey, a teacher would be able to identify seven distinct mathematics strands or dimensions: Numeration and Number Theory, Whole Number Operations, Fractions and Decimals, Measurement/Time/Money, Geometry, Probability/ Statistics, and Pre-algebra. The test specifications for that exam imply that the mathematics test questions are primarily composed of problem-solving tasks. Therefore, it is safe to assume that test questions will require thinking in the application, analysis, and perhaps synthesis and evaluation levels of cognition.

Once teachers and administrators specify all of the subject area dimensions, the following activities can begin:

* * Selecting and designing classroom assessments and practice questions.

* * Revising and designing curriculum that is congruent with the content identified in the state standards and the district's delineation of the state-designed exams

* * Designing teacher training using instructional techniques that support these dimensions.

ALIGNMENT

During the alignment phase, administrators and teachers work to identify, analyze, generalize, and describe the links between the various elements associated with the subject area previously delineated and the sample questions selected for practice or classroom activities to assess student progress. The sample questions and student assessments can be derived from several sources, including state-released test items, commercially manufactured test preparation materials, or teacher-made activities. Teachers and administrators examine linkages in the materials, organization, textbooks, instructional strategies and other elements described in the curricula and used in daily instructional activities to ensure consistency with the district's delineation of the state assessment.

Using and understanding the test specifications become even more important at this stage. Returning to the example above, a teacher seeking to ensure that students would be able to understand tree diagrams and solve problems using them would complete several alignment tasks:

1. Review classroom resources, curriculum, textbooks, teacher activities, student thinking strategies, and tests to ensure that the test specifications and macro standards are addressed on the knowledge and problem solving level. Do the teacher resource materials and classroom instruction address the proper skills?

2. Review the above factors to ensure congruency between the level o difficulty required by the standards and specifications, and the difficulty of the actual teacher resources and activities. Do the teacher's tests, lessons, and activities match the difficulty level required by the standards and specifications?

3. Consider format. Although less important than skills and difficulty, the teacher resources, activities, and tests should familiarize the students with state test question formats. Teachers must align classroom assignments and activities to the subject area delineation to ensure congruency.

CALIBRATION

After completing the delineation and beginning the alignment processes, calibration begins. Calibration is the act of conducting communications and interactions with teaching staff based on the information identified in delineation and used in alignment. The calibration process ensures that the conceptualization of content, cognitive process, complexity, formats, etc., is consistently understood for each subject area. Calibration, in its simplest form, is designing classroom instruction, activities, and assessments that are congruent with content area delineation and alignment. Using the mathematics vignette as an example, one can begin to see how the process takes place.

Figure 1. Represents the sequence of events leading up to Calibration.

* Delineation

1. Teacher performs content area delineation and chooses a unit of focus. In this case Standard 4.1, Cluster IV-Discreet Mathematics-tree diagrams

* Alignment

2. Teacher examines a compares classroom resources, local curriculum, activities, skills level of difficulty, format, and tests to the Standards and test specifications to ensure congruency.

* Instruction

3. Teacher uses and implements calibrated activities, tasks, resources and lessons.

* Calibration

4. Teacher designs lessons and activities, gathers resources, and creates tests that are congruent with the skills and level of difficulty for the Standards, test specifications and curriculum.

Figure 1. Delineation, Alignment, and Calibration Flow of Events

Matt has four channels on his television. He has channels 2,3,4 and 5. If Matt watches only two channels each night, haw many different combinations of channels can he watch? Show all your work, and explain your answer. Matt can watch (2,3), (2,4), (2,5) or (3,2), (3,4), (3,5) etc.

A 4th grade teacher completing delineation and alignment and discovering that her/his program was missing a unit on discrete mathematics would develop objectives related to understanding, using, and interpreting tree diagrams. Figure 2 is a sample activity/test question created by 4th grade teacher Terry Maher to begin addressing the aspect of discrete math noted in the Cluster IV test specification.

Calibration is any action that helps teachers design activities and construct assessments based on the dimensions of state assessments and standards. This process helps to foster a collective understanding and agreement of the dimensions and domains of each content area. It should be a team effort based on group inquiry.

USING SCORE REPORTS TO IMPROVE CALIBRATION

As teachers gain a better understanding of how student work reflects the standards and test specifications through delineation, alignment, and calibration, their efficiency and accuracy at identifying which students are meeting the standards should increase. Herein lies the usefulness of score reports sorting students into categories of varying proficiency. A student who scores partially proficient, proficient, or advanced proficient on a state language arts test may also show some congruency in the level of achievement in his/her well-aligned school work and classroom assessments. As teachers become better calibrated, they will be able to answer questions such as: What level of proficiency does this student show on class assessments? Is the difficulty level of the class work comparable to the state exam? What can I do to help this student meet the state standards? Is my program meeting the standards?

PREDICTING OUTCOMES

Teachers can reflect upon their level of calibration accuracy by attempting to predict student results on state assessments. Teachers should be aware, however, that 100% agreement should not be expected between a student's performance on well-calibrated classroom tests and on the state assessments, based on many factors of test design.

To begin the prediction process, the teacher uses a list of the students taking the test. Beside each name, the teacher enters a predicted score level. When the state assessment scores arrive, the teacher can compute the level of accuracy as shown below.

Name Prediction Score

Allan Proficient Adv. Proficient

Ann Proficient Proficient

Tamika Adv. Proficient Proficient

Bronson Partial Proficient Partial Proficient

* The list above shows a 50% level of success in the predictions made.

The teacher making the predictions returns to each student's work and compares the successful predictions with the unsuccessful ones to gain a better idea of how the assessment performances reflect the aligned student work. Student work associated with actual test scores can form the basis for subsequent calibration discussions. Student work connected to state assessment score levels can also function as scoring examples that students refer to when judging their own work.

FINAL THOUGHTS

There is a distinct difference between traditional notions of test preparation and aligning and calibrating instruction and assessments with the content, cognition, difficulty, and format of state assessment instruments, specifications, and standards. As educators, we are trying to link the classroom activities to the standards and skills set by the state. The aim is to ensure that teachers understand, and calibrate their classrooms with respect to, the entire process and do not simply focus on how to answer specific types of test questions. The questions will change, but the underlying skills and concepts will not. One must be careful not to wallow in the mire of test prep.

Delineation, alignment, and calibration are academic endeavors that demand unending commitment. Do not expect to accomplish alignment or calibration at an in-service day, or even during the course of a school year. The administration must provide the time and resources to conduct frequent calibration meetings to examine such things as classroom work and student assessment samples. Beware, it is easy to fall out of alignment and calibration and into test prep.

REFERENCE

Black, Paul, and Wiliam, D. ( 1998 ). Inside the black box: Raising standards through classroom assessment, Phi Delta Kappan October, pp. 139-148.

-----

This publication was prepared with funding from the Office of Educational Research and Improvement, U.S. Department of Education, under contract ED99CO0032. The opinions expressed in this report do not necessarily reflect the positions of policies of OERI or the U.S. Department of Education. Permission is granted to copy and distribute this ERIC/AE Digest.


Title: State Standards, Assessments and Instruction. ERIC Digest.
Document Type: Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073);
Available From: ERIC Clearinghouse on Assessment and Evaluation, 1129 Shriver Laboratory, University of Maryland, College Park, MD 20742-3742. Tel: 800-464-3742 (Toll Free). Web site: http://ericae.net.
Descriptors: Educational Assessment, Educational Improvement, Elementary Secondary Education, Information Dissemination, Instruction, State Standards, Test Construction, Test Use
Identifiers: Calibration, ERIC Digests, New Jersey, Test Specifications

###

[Return to ERIC Digest Search Page]