Introduction

This report summarizes the changes in student achievement scores from 313 elementary (E), junior high/middle (JM) and high (H) schools that completed an improvement cycle and were accredited by NCA in the academic years 1999-2000 and 2000-2001. The purpose of this analysis is to continue to chart the progress of the NCA CASI schools in their improvement efforts. The present report extends the reporting sequence initiated by Wick (1997) and Armstrong (1998, 2000). Primary emphasis in the current report is on changes in student achievement (expressed in adapted standard units for easy comparison) and on patterns and trends in goal selection.

There were some differences in the format of the reporting between the 1998 and 2000 reports that made certain comparisons difficult. Therefore, the format this report is patterned precisely on that of the 2000 report, thus making direct comparisons more convenient and rendering trends more easily noted. (See the next section for a brief explanation of standard score conversion and score assignment for goals and for schools.)

General Pattern of the Improvement and Documentation Process

Although NCA CASI endorsement nomenclature has changed over time, the general pattern of the improvement process has remained relatively stable. A school begins an improvement cycle by conducting an information scan of students/school/community to develop what is called a "school profile." The purpose is to gather and analyze data about student learning and learning needs, instructional practices, and school and community resources so that a purposeful and comprehensive improvement plan can be developed. An analysis of the school profile, combined with informed input from other sources, leads to identification of two or three major goals.

As soon as goals are identified, the selection and/or creation of assessments begins, simultaneous with the task of developing a comprehensive improvement plan that is designed to nurture improvement of student achievement in all of the selected goals. Immediately prior to the implementation of the improvement plan, the appropriate "pretesting" is conducted. This so-called pretesting is actually a form of posttesting, assessing the learning of all the students in the school (or all the students at one or more selected grade levels) on the subject matter of the selected goals. These are the students who have been taught the subject content of the goals in the traditional manner. These "pretest" score data are the baseline for what amounts to an educational experiment.

Immediately following the baseline testing, the improvement plan interventions are initiated. This period of implementation normally lasts for three academic years. At the completion of the improvement cycle, posttesting is conducted on all students or on the same grade level(s) of students that were used to collect baseline data. The baseline and posttest instruments must be the same, or psychometrically equivalent, so that score comparisons are valid. Posttesting is conducted at the same time of the academic year as the baseline testing was done. The difference scores (contrasting baseline and post scores on each assessment) are expressed in adapted standard units and help the staff members analyze the effectiveness of the improvement plan interventions as compared to the traditional teaching methodology.

At the completion of the full improvement cycle (normally five years) the school staff summarizes the improvement effort, including change-of-achievement results, in a final documentation report. If the combined process and/or change-of-achievement numbers are considered satisfactory by the visiting team, the state committee recommends accreditation. The present report was assembled from change-of-achievement scores in the final documentation reports of 313 schools that completed their improvement cycles and were accredited in the academic years 1999-2000 and 2000-2001.

In each school report, a standard score was computed for a goal if at least one valid assessment yielded a numerical score difference between the baseline assessment and posttest. In the case of multiple assessments for one goal, the score differences in standard units were averaged for the goal score, unless results were so discrepant that at least one assessment was clearly not aligned with the goal objectives. Then, either the offending assessment score was removed, depending upon the school staff judgment, or the score results on that goal were omitted. For a given school, an overall change-of-achievement score in standard units was assigned if at least half the goals were assigned a score. The average (arithmetic mean) of all goal scores yielded an assigned score for the school.

Data and Commentary

For the presentation of data, the same format is presently employed as was used in the 2000 report so that direct trend comparisons can be made. For this purpose the reader might review the Spring 2000 issue of the Journal of School Improvement and follow the "Change of Achievement" article when reading the present article. In the present report, unit schools were omitted because their number was too small for meaningful analysis.

In Table 1 the average standard unit change of achievement is shown for each school level. Also exhibited in Table 1 are the number of schools at each level, the number of schools receiving no change-of-achievement score, the number receiving a negative score, and the high and low school scores at each level.

Table 1

Data by School Level

The change-of-achievement scores (second data column) are the main focus of Table 1. These scores are "adapted standard units" and are equivalent to standard deviations. A school with a gain score of .25SU(standard units) has shown an average increase in student achievement on the selected goals of a quarter of a standard deviation, equivalent to approximately 10 percentile points in the normal distribution midrange.

Referring to column three, a school is assigned an adapted standard score if at least half of its goals for that improvement cycle were assigned adapted standard scores, in which case the school score is the average (arithmetic mean) of the goal scores. In column four we see the number of schools that achieved a negative score. This indicates that the students in those schools experienced less achievement in the improvement cycle using new methodologies compared to the baseline students, who were taught by the traditional methodology.

For the purposes of value estimation and interpretation of school scores, an arbitrary but empirically plausible set of division points has been adopted to allow the use of descriptors for school scores. These guidelines are provided to help determine if the achievement changes resulting from the interventions of the improvement plan are large enough to be meaningful and useful. The division points and suggested descriptors are, as follows:

Due to normal test and aggregation error, the range from -.09SU to .09SU is viewed as indicating neither an increase nor a decrease of the improvement cycle scores when compared to the scores of the traditionally taught students. For the distribution of school scores over the above descriptive range, see Table 2.

As noted, if fewer than half of the school's goals were assigned a score, the school would receive no overall score (Table 1, third data column). In this two-year period sample, 11 schools (less than 4%) received no score. A negative score (fourth data column) indicates that student change-of-achievement using new methodology was less than using the traditional methodology. Such results need to be carefully examined, however, since the phenomenon of entire class levels varying from other levels in academic ability or preknowledge is well known to teachers, particularly in the elementary schools. In the present report, 30 schools (less than 10%) reported a negative score. Of these, 11 scores were below -.09SU, indicating that 11 schools may have had achievement results verifiably lower than from the traditional methodology.

Comparing the data in the second column to the previous two reports by Armstrong (1998, 2000) we see that the change-of-achievement score for all schools increased slightly to .27SU, over .26SU in the 2000 report and over .23SU in the 1998 report. As compared to the three school levels in the 2000 report, the elementary score dropped slightly, from .28SU to .26SU, but both junior high/middle and high schools increased, .22SU to .27SU and .25SU to .28SU, respectively.

A somewhat different perspective on school scoring results can be seen in Table 2. Here the adapted standard scores are shown distributed over the descriptive range noted above. The two rows below "Total number of schools" yield a comparison between currently reported schools and those of the previous two reports (1998, 2000). It can be seen that for schools scoring less than .09SU (no verified change-of-achievement, or negative change-of-achievement), the current set of schools fits precisely between 1998 and 2000. The current set of schools has fewer in the ".10 to .19" range but substantially more in both ".20 to .29" and "S>.30," which supports the overall increase in the mean school score noted in Table 1. In all, it may be noted that 78% of the schools in the present report had "verifiable" student achievement gain.

Table 2

Goal Selection

A look at goal selection, and an alternative view of achievement by goals, is offered in Tables 3, 4 and 5. Table 3 exhibits the high school goal selection pattern. It is apparent that there continues to be a heavy choice in the basics: reading, writing, and mathematics computation. In fact, the proportion is up five points, from 54% in 2000 to the present 59%. Further, critical-thinking and problem solving, combined, are down seven points from 30% to the present 23%. However, the basic skills combined with critical-thinking and problem-solving still dominate the cognitive goal choices, accounting with 82% of all high school goals being in the category. In the current report, technology goals continue to be seldom chosen with study skills disappearing completely. It may be that most schools are simply treating technology competence and study skills as the means for learning other things, rather than as ends in themselves. The proportion of goals marked "other" has nearly doubled but encompasses too many unique and varied choices for meaningful analysis.

Table 3

As for affective goals, the primary trend is further shrinkage of respect, self-esteem, and "other" with the slack being taken up by responsibility. Whereas responsibility constituted slightly less than 50% of the affective goals in the 2000 report, in the present report the proportion is 66%. If cognitive goal choices are, at least in part, influenced by state directives, the affective choices are essentially unstructured, leading to the inevitable conclusion that student responsibility is a growing concern. Are schools seeking authentically responsible behavior or simply compliant behavior?

The reader might note that for high schools a higher percent (91%) of affective goals had numerical scores that could be converted to standard units than did the cognitive goals (89%). Given the greater difficulty most schools encounter in measuring affective goal results, as compared to cognitive goal results, this is unusual. It is an exception to both elementary and junior high/-middle and high schools in this report, and to all school levels in the 2000 report.

Table 4 shows the same information for junior high/middle schools. The distribution patterns for both cognitive and affective goals are similar to those of the high school. However, the concentration in the basic skills is even more pronounced (67%), as is the concentration in the goal area responsibility (75%). Presumably, the same forces (the state and social contingencies) are at work in the junior high/middle school as in the high school.

Table 4

In the lower part of the table it should be noted that the mean value of affective goals is .94. This means, of course, that some schools have opted not to address an affective goal, at all. It might also be noted that for both high school and junior high/middle school, the total number of goals selected for improvement cycles is shrinking. This trend may represent a degree of realism developed from the tendency of earlier schools to almost literally "bite off more than they could chew" when planning an improvement cycle. Current NCA CASI inservice preparation for school improvement now recommends two or three major goals.

The distribution pattern of goal selection for elementary schools is shown in Table 5, and is also similar in distribution to the junior high/middle and high school patterns. The trend of higher concentration of cognitive goals in the basic areas (math, read, write) continues with elementary schools, with a total of 72%. "Responsibility" as an affective goal increased over the 2000 report, with 81%. Relatively fewer elementary schools selected an affective goal than did even the junior high/middle schools.

Table 5

Trends and Discussion

Comparing all tables to their counterparts in the 2000 report and to selected data in the 1998 report, a few trends or conclusions seem applicable.

The trends in goal selection are interesting but difficult to interpret in some cases. The growth in selection of "responsibility" as a goal is so great that the concern of school faculties for this student attribute seems obvious, regardless of a given school's precise definition of "responsibility." Less obvious are the reasons for the decreasing number of schools that are selecting an affective goal, at all. It is true that affective traits are generally more difficult to define and are, hence, more difficult both to nurture and to assess. However, there is ample evidence in the literature attesting to the critical importance of the affective constructs in human lives.

The popularity of the basic skills in the cognitive area continues, with a modest increase in mathematics computation. However, there was a sharp drop in both critical-thinking and problem solving. This fact, combined with the reduction in the selection of affective goals, seems to support the notion that schools may be tending to shy away from goals that are more difficult to define and, hence, more difficult to cultivate and to assess.

Concluding Comments

The general structure of the NCA CASI school improvement process has been in place in its essentials since 1984, with the first 11 schools receiving "outcomes" endorsement in 1987. Thus, assuming an average improvement cycle of five years since that time, we are now near the end of the third generation of our basic improvement process. Based on the sequence of the four noted reports, including the present one, both goal and overall school scores in adapted standard units have remained stable or improved over this time. It seems apparent that the core ingredients of the process offer a format for a reasonable expectation of improvement in student achievement if this improvement effort is undertaken intelligently and in good faith with the appropriate support of the administration and community. These core ingredients are, as follows: