NAEP Large City Study Sheds Light on the Effects of the Atlanta Public Schools’ Cheating Scandal.

The National Assessment of Educational Progress (NAEP) created the Trial Urban District Assessment (TUDA) in 2002 to assess student achievement in the nation’s large urban districts.  Reading results were first reported in 2002 for six districts, and math results were reported in 2003 for 10 districts.

The NAEP provides data from 2002 through 2012 on math and reading and are comparable to NAEP national and state results because the same assessments are used.

In July 2011, the Governor of Georgia released a report of its investigation into the Atlanta Cheating Scandal, charging 178 educators as being involved in the scandal.  According to the report, thousands of school children were harmed by widespread cheating in the Atlanta Public Schools (APS).

According to the Governor’s report, “a culture of fear and conspiracy of silence infected this school system, and kept many teachers from speaking freely about misconduct.” Although I’ve never condoned the cheating that occurred in the APS, the report falls short by not pursuing what caused the culture of fear to exist in the system, which apparently led to the cheating.  Who, besides the employees of the APS were involved in the so-called conspiracy? What role did the following play in this scandal:Georgia Department of Education, the Governor’s Office of Student Achievement, the Atlanta School Board, and partners who contributed millions of dollars to the APS to boost academic achievement of Atlanta’s students.

According to the report, the cheating scandal took place in 2009.  By 2010, the scandal had been exposed by the Atlanta Journal and Constitution reports on the APS, and we can assume that there was very little, if any cheating on the state’s 2010 – 2013 Criterion Referenced Competency Tests (CRCT).

During the period leading up to, during, and after the cheating scandal, the NAEP tested students in Atlanta, as part of the Trial Urban District Assessment in mathematics and reading from 2002 to 2012.  Fourth and eighth grade students were tested using the NAEP tests.

Don’t you think that examining the data on the NAEP tests given as part of Trial Urban District Assessment might be helpful in several areas?

• What is the trend of academic performance of Atlanta students (grades 4 and 8) in mathematics and reading during 2002 – 2012?
• Are there significant changes (increases, decreases) or no changes in the Atlanta data during this period?
• Is there evidence that the academic performance of students in the APS was harmed or diminished during and after the scandal?  Do student scores change appreciably after we can be sure that there was little if any cheating going on?  Were students victimized as a result of the testing scandal?

What is the trend of academic performance of Atlanta students (grades 4 and 8) in mathematics and reading during 2002 – 2012?

Figure 1 summarizes Atlanta eighth grade student scores on the NAEP test in mathematics given as part of the Trial Urban District Assessment.  I’ve plotted the average scores of students at the 25th-50th-75th percentiles. The tend at each level is up and there is no evidence here of a decline or slump in scores for 8th grade students.

The Atlanta NAEP scores in 2011 and 2013 did not decline following the 2009 cheating scandal.  This is an important finding in the context of the Atlanta Cheating Scandal.  If students had been harmed academically, then their scored might have dropped after the episode of cheating. For more details that go beyond the graph that I produced here, consult this page in the TUDA 2013 report.

Figure 1 is a graph showing the average score of students at the 25th, 50th and 75th percentiles. NAEP, TUDA 2013 Report

What’s interesting in the data are the scores in 2011.  The 2011 eighth grade students were in the sixth grade during the year of the cheating scandal.  If the students were academically victimized because of changes in CRCT scores, then we would hypothesize that their scores would decrease in the 2011 and 2013 years.  But they did not.  In fact, there is an increase in the scores at each percentile level.

Are there significant changes (increases, decreases) or no changes in the Atlanta data during this period?

Figure 2. NAEP Math scores for APS 8th grade students and large city districts.

The scores of Atlanta eighth grade students are plotted and compared to the average scores of other large cities that participated in the NAEP Trial Urban District Assessment.  Although Atlanta’s scores are lower than the average for each year, the overall trend is upwards, and the gap is closing.

Again, when we compare the years following 2009, the year of the test erasure scandal, two and four years later, Atlanta students are not only doing better, but they are closing the gap.  Is there evidence here that students were academically harmed by the scandal?

Is there evidence that the academic performance of students in the APS was harmed or diminished after the scandal?  Do student scores change appreciably after we can be sure that there was little if any cheating going on?  Were students victimized as a result of the testing scandal?

The NAEP data is separate and administered differently than the state CRCT tests.  Indeed, the NAEP tests are low-stakes, and according to many researchers,  NAEP scores are more valid and reliable than the high-stakes CRCT if one wants to have an idea about student performance.  CRCT is a high-stakes assessment that is used not only to assess the students, but the results are used to evaluate teacher performance.

There clearly are reasons to wonder if students were harmed by the cheating that took place in 2009 in the APS.  Academically, there is little evidence based on average scores reported in the NAEP study.  However, we have to wonder what was the social-emotional consequence caused not only by the cheating that took place, but also by the standardization and high-stakes testing reform movement that most likely what contributed to the “culture of fear and conspiracy” in the APS.

Stephanie Jones, professor of education at the University of Georgia has written extensively on the social-emotional consequences of the authoritarian standards and high-stakes testing dilemma.  She asks, “What’s the low morale and crying about in education these days?  Mandatory dehumanization and emotional policy-making–that’s what.”

Policy makers, acting on emotion and little to no data, have dehumanized schooling by implementing authoritarian standards in a one-size-fits-all system of education.  We’ve enabled a layer of the educational system (U.S. Department of Education and the state departments of education) to carry out the NCLB act, and high-stakes tests, and use data from these tests to decide the fate of school districts, teachers and students.  One of the outcomes of this policy is the debilitating effects on the mental and physical health of students, teachers and administrators.

If you don’t believe that, here is a quote from Professor Jones’ article:

I’ve witnessed sobbing children in school, tears streaking cheeks. When children hold it together at school, they often fall apart at home. Yelling, slamming doors, wetting the bed, having bad dreams, begging parents not to send them back to school.

More parents than ever feel pressured to medicate their children so they can make it through school days. Others make the gut-wrenching decision to pull their children from public schools to protect their dignity, sanity and souls. Desperate parents choose routes they had never thought they’d consider: home schooling, co-op schooling, or, when they can afford it, private schooling. But most parents suffer in silence, managing constant family conflict.

 Were Atlanta’s Students Harmed by the Test Erasure Scandal?

Based on NAEP data, Atlanta students continued to improve in mathematics, even after cheating was discovered, and eliminated from the district.  Although NAEP does not investigate the social-emotional effects of school, there is evidence that the current emphasis on high-stakes testing does contribute to and has amplified emotional and behavioral disorders among youth.  How can it be a good practice that in one district in Georgia, 70 of 180 days of the school year are devoted to some kind of state or federal testing?  For more data on this, please refer to a discussion of The Paradoxes of High Stakes Testing by Madaus, Russell, and Higgins (2009) in this blog article.

The NAEP large cities study does shed light on the Atlanta Public Schools.  There is evidence that any harm that directed toward students was more psychological than academic.

In spite of the national and international attention that the testing scandal generated, teachers in the Atlanta Public Schools positively impacted their students in mathematics and reading.  There is evidence if you dig deeper into the data that there is a continued need for more resources and more experienced teachers in schools which are populated students living in poverty, and students who are on free or reduced lunch.