The Numbers Game
By Deborah Jeffries Dusseau, David S. Hurst, and David Bitter

Excerpted and adapted with permission from: Dusseau, Hurst, and Bitter, "The Numbers Game" Principal Leadership, September 2003, p. 49-54. Principal Leadership is a publication of the National Association of Secondary School Principals (NASSP). For more information concerning NASSP services and/or programs, please call (703) 860-0200, or visit http://www.principals.org. For the full article visit http://www.ncacasi.org/documents/assess_game_prin_leader.pdf.

What Game Are You Going to Play?

An early consideration in the assessment game is to be clear about what you plan to measure. Before a valid and productive assessment program is developed, you need to know what game you are playing - the cohort growth game or the program growth game?

Assessing program growth yields different information than cohort growth and may be easier for educators to design, implement, and understand at the school level. Educators often complain that many of the outcomes for which they are held accountable are beyond their control. Educators don't control the ability levels, numbers, family backgrounds, level of parental support, or native languages of the students who come to their classrooms. In other words, educators do not control the cohort.

Conversely, what happens within the classroom - the educational program - is largely under the control and influence of educators. The educational experiences the students engage in, the teaching techniques used, the timing and pace of the curriculum, and how learning is practiced and reinforced are largely controlled by the instructor. Program growth focuses on the impact of these variables.

What are the Rules of the Game?

School accredited by the North Central Association Commission on Accreditation and School Improvement [NCA CASI] have been focusing on program growth measures for the last fifteen years as part of performance based accreditation. While implementing this protocol, NCA CASI has identified some rules of the game that we call Green Light and Red Light Conditions.

How Will You Know If You Won?

After you have determined exactly what you want to measure and considered the rules for evaluating your assessments, how will you know if you are truly improving? Two major issues come to mind: 1) multiple assessments provide a more comprehensive picture of student performance and 2) interpretation of results from multiple assessments requires the use of specific statistical tools.

Use Multiple Assessments. Bernhardt, in her book Data Analysis for Comprehensive School Improvement, advocates that ". . . multiple measures must be considered and used to understand the multifaceted world of school from the perspective of everyone involved. . ." (1998, p.13). Sergiovanni (2000) recommends multiple assessments that reflect both state and local standards. The conclusion is that we must rely on more than one source of data to substantiate that student performance, in the overall sense, is improving.

Convert Results to Standard Scores. If a person accepts the premise that multiple measures are preferable over a single assessment, then the "game" gets complicated. If one assessment uses percentile scores as a metric and another uses raw scores, how can both assessment results be used and compared? One relatively simple answer to this problem is to compare standard scores, or z scores. Simply defined; "a z score is expressed as units of standard deviation above or below the mean" (McMillan, p. 116). By determining the difference between a pre-test z score and a post-test z score, we can measure the growth in standard deviations-a "common denominator" for statistics. This difference is known as effect size, or the magnitude of change.

Calculating z scores and effect size is a relatively simple task. There are several software packages available that can calculate effect size simply and quickly. Most statistics textbooks provide tables that allow standard scores to be calculated with little stress.

Even after data are converted to standard units, a major question remains: Is the growth worth mentioning? Armstrong (2002) conducted a multi-year study of 600 NCA CASI accredited schools that used adapted standard scores to measure the results of their school improvement activities. Based on that study, Armstrong concluded that an effect size or standard unit growth of 0.3 was indicative of substantial growth, growth of .2 to .29 standard units was considered good, and growth of .1 to .19 standard units was worth mentioning. Declines in performance were defined by the same scale. Readers should realize that effect size (or standard unit growth) is a subjective term and should be defined in the context of the study.

Summary

The decision to document program growth, cohort growth, or both should be based on an honest representation of the data in context rather than being based on pressure to achieve a "winning score." If the assessment design was compromised, the final score lacks meaning and credibility. Finally, the use of standard scores allows for accurate interpretation of multiple assessment results, thus reducing dependency on single "high stakes" assessments. As in all sports, there is more to this assessment game than the final score.

Dusseau, Hurst, and Bitter are staff members at NCA CASI. To read their full article and see the list of references, please visit http://www.ncacasi.org/documents/assess_game_prin_leader.pdf.

Resources - Useful Data Analysis Resources

NCA CASI Software
Available free to NCA CASI accredited schools, this data analysis software helps schools disaggregate, analyze, monitor, and document student performance data, as required by NCA CASI and NCLB. Accredited schools can download the software at http://www.ncacasi.org/sitools/software.

Multiple Measures: Accurate Ways to Assess Student Achievement
By: Joan Ardovino, John Hollingsworth, and Silvia Ybarra
Available at: http://www.corwinpress.com/book.aspx?pid=5083
This practical guide for educators and administrators shows how to measure teaching and learning accurately with multiple measures. It includes strategic handouts, reports, graphs and charts, and communiqués that will help you create an effective assessment program and meet federal requirements.

Essential Assessment Concepts for Teachers and Administrators
By: James H. McMillan
Available at: http://www.corwinpress.com/book.aspx?pid=5029
This book covers the following: the role of assessment in teaching, how validity is determined, factors influencing reliability, fairness in avoiding stereotypes, understanding and using numerical data, using standardized assessment results to improve instruction, practical suggestions for classroom assessment, and data analysis for comprehensive school improvement

Data Analysis For Comprehensive Schoolwide Improvement
by Victoria L. Bernhardt
Available at: http://www.amazon.com
This useful book provides information on disaggregating data, assessing perception, assessing performance, assessing school processes, communicating results, and using data to improve student and school performance.

The Memory Jogger for Education: A Pocket Guide of Tools for Continuous Improvement in Schools
Adapted by Ann McManus
Available at: http://www.goalqpc.com
This pocket guide is designed to help public and private educators at elementary, secondary, and post-secondary levels to improve on a daily basis the procedures, systems, quality, cost, and outcomes related to the educational process.

Commission Corner

NCA CASI District Accreditation Demonstration Sites

Three districts have been selected as demonstration sites for implementation of NCA CASI's new District Accreditation. They are:

- Cuba Independent Schools, Cuba, NM
- Dublin City Schools, Dublin, OH
- Hutchinson Public Schools, Hutchinson, KS

The districts will be receiving in-depth training on system improvement throughout the year. As they gain experience in the district accreditation framework, they will become the "go to" districts when new districts have questions.

To learn more about District Accreditation, visit http://www.ncacasi.org/standard/district or attend a District Accreditation Orientation workshop (see dates in the Professional Development section).

Annual Report Deadlines

Just a reminder for accredited schools: The demographic portion of the NCA CASI Annual Report is due during the October-November timeframe for all elementary, middle level, secondary, unit, college preparatory, and adult/vocational accredited schools. Special purpose and postsecondary schools complete their reports later in the year on a different timeline. Following, please find a list of deadlines for each state.

Assessment Questionnaire

We want your feedback! How can we assist you through enhanced assessment services? We want to help you use assessment instruments and data effectively and appropriately, and help you link NCA CASI accreditation, state, and federal assessment requirements. Please take a few minutes to complete the brief assessment survey. Your assistance will be greatly appreciated.

Annual Meeting Notes
NCA CASI 109th Annual Meeting
March 28-31, 2004 - Chicago, IL

Join us for three days of discussion, exploration, and sharing of practices and processes that result in improved student learning. Listen to keynote speaker, Dr. Lorraine Monroe of the Lorraine Monroe Leadership Institute share "How to Create Effective Schools for All Children." Join Tom Guskey, university professor and assessment expert, as he shares tips for using research in efforts to close the achievement gap. Choose from more than 100 practitioner-led breakout sessions as well as several in-depth seminars designed to provide you with proven strategies for raising student achievement at the classroom, building, and system levels. Visit http://www.ncacasi.org/event/meeting to find out more.

Upcoming Professional Development

To find out more information or to register for the following workshops, visit http://www.ncacasi.org/eventsandtraining.

Start the New Year in the School Improvement Specialist Program!

How to Reach Us

We are committed to providing you with the information you need to continually improve student learning. Please share with us your suggestions, advice, and ideas on how to make e-News and our other products and services best meet your needs. Send us feedback at enews@ncacasi.org.

Thank you for reading this issue of NCA CASI e-News. To see a copy of this newsletter on-line or to view past issues of e-News, go to http://www.ncacasi.org/enews/index. Please report problems to enews@ncacasi.org.

North Central Association
Commission on Accreditation and School Improvement
P.O. Box 874705
Tempe, AZ 85287-4705
800-525-9517
http://www.ncacasi.org

Thank you for reading this issue of NCA e-NEWS. Events and dates are subject to change.
Please report problems to enews@ncacasi.org. To see a copy of this newsletter on-line, click here.