Latest Story

People are asking for better schools, with no clear idea how to improve eduction, or even how to define improvement of education (except to increase test performance on high-stakes tests).

Most people are in favor of improving education.  But when asked how would they improve education, the suggestions are insufficient, and in some cases, even negative (See W. Edwards Deming. The New Economics for Industry, Government, Education (p. 8). Kindle Edition.)

Instead of reporting the details of how people and organizations want to improve education, such as corporate chiefs, philanthropists, and the U.S. Department of Education, I want to report on the work of Mr. Ed Johnson, an advocate for quality education, and has for more than a decade devoted himself to writing and talking about improving education in the Atlanta Public Schools.

Systems Thinking

Ed Johnson consults as Quality Information Solutions, Inc., with a commitment to human social and cultural systems to receive quality information from information systems for the continual improvement of life, work, and play. His commitment extends to advocating the transformation of K-12 public education systems to humanistic paradigms from prevailing mechanistic paradigms. Ed also is former president of Atlanta Area Deming Study Group.

Ed Johnson is a systems thinker.

In this regard, he believes schools can not be improved by trying to improve the parts separately.  It is a sure path to failure.  For example, some advocates of educational reform believe that student achievement can be improved by weeding out the bad teachers.  Millions of dollars have been invested in using student high-stakes test scores to check teacher performance using a technique called Value Added Measure (VAM).  Teachers whose VAM scores are low can be identified, and according to these experts, teachers with low scores must be bad teachers.  Getting rid of “defects” in any system will not improve the system or the part that was identified.  Instead, a better investment would be to ask how can we improve the quality of teaching, and what can be done to improve the teaching of all educators.

The above example highlights the current approach to reform.  Identify a part of the system, and fix it. Bad teachers, get rid of them.  Low achievement scores?  Write “rigorous” standards, raise the bar, and give high-stakes tests.  It’s that simple.  We’ve had rigorous and not so rigorous standards in place for more than a decade, and as you will see ahead, changing standards doesn’t have any effect on student performance.

Systems thinking means that all parts of a school system are interdependent and must be taken as a whole.  The Atlanta Public Schools (APS) is a system of interconnected and interdependent parts, and to improve the quality of the APS, it is critical to look at the APS as a whole.  For example, closing schools (removing so-called underperforming schools), does not have an effect of improving the APS, or indeed saving money (as some would tell you).  Fundamental questions about APS need to be asked, but in the context of the APS being a system, not a collection of  schools, students, teachers, administrators, parents, curriculum, textbooks, technology.

Ed Johnson has contributed to my understanding of quality education, and it is my great honor to share his work on this blog, and in particular to look at teaching in urban schools, and in particular the Atlanta Public Schools.

Trial Urban District Assessment (TUDA)

The National Assessment of Educational Progress (NAEP) created the Trial Urban District Assessment (TUDA) in 2002 to assess student achievement in the nation’s large urban districts.  Reading results were first reported in 2002 for six districts, and math results were reported in 2003 for 10 districts.

The NAEP provides data from 2002 through 2012 on math and reading and are comparable to NAEP national and state results because the same assessments are used.

Using data from all of the Trial Urban Districts Assessments (2002 – 2013), available online at The Nations Report Card, Ed Johnson analyzed and created a presentation that is a series of systemic stories told by the data collected in the urban district studies.

Each story is about a system.  In TUDA (Trial Urban District Assessment), there are particular TUD’s each as a system.  So, Reading is a system.  Mathematics is a system.  4th grades is a system.  8th grades is a system.

Johnson’s research is a longitudinal study of performance in reading and mathematics from 2002 – 2013.  Using scores on reading and mathematics obtained from the National Center for Educational Statistics, he investigated the nature of a number of systems derived from the data.  Some of these particular systems include:

• Reading as a System
• Mathematics as a System
• 4th Grades as a System
• 8th Grades as a System

Since these systems are part of the APS system, we know that each of these systems is interdependent with other systems, not just the ones identified here, but including parents (as a system), teachers (as a system), and so forth.

There were 21 urban school districts in the study.  However, Ed has managed to make our work easier by highlighting with color coding just two systems, Atlanta Public Schools (Red) and District of Columbia (Purple).

Overall

Figure 1. Trial Urban District, Bottom Line. Source: Ed Johnson edwjonhson@aol.com

Ed starts his study by giving us the “bottom line.”  How did these systems (reading, math, 4th grade, 8th grade) do?  Figure 1, is a summary of systemic TUD student performance in reading and math at the 4th and 8th grade from 2002 – 2011, and predictions for 2013 (all the predictions were accurate forecasts of student performance in 2013).

Only 4th grade reading showed some improvement over the period 2002 – 2011, and the improvement was slight and noticed only in Austin, Charlotte, and Hillsborough.  In all other systems, no improvement was observed, meaning that the common causes that influence the system of math, or reading, or 4th or 8th grade inhibited improvement.

Student Improvement in Mathematics and Reading

In the TUDA study, a sample of students in each urban district was tested in reading and mathematics at the 4th and 8th grade level.  To help us understand how to interpret data collected over the past dozen or so years, Mr. Johnson has produced a series of graphs (control charts) showing the natural variation of scores to be expected in each system (reading, math, 4th grade, 8th grade).

Figure 2 shows a control chart for  reading, 4th grade.  Figure 3 shows a control chart for mathematics, grade 4.  Upper control limits and lower control limits were calculated for 2002, and then projected forward.  Changes in scores from one test period to the next are shown in the Figure 2.  If there is systemic change in reading at the 4th grade level, then scores would fall “outside” the upper or lower control limits.  You’ll notice that all the variation, except for four points (Charlotte, 2009 and 2011, Austin, 2011, and Hillsborough, 211), was within the variation expected.  In systems thinking, we mean that the variation for the most part was random, but there is evidence that some special causes were at work in the three districts mentioned here.

Figure 2. NAEP TUDA, Reading, 4th Grade, All students prepared by edwjohnson@aol.com

Mathematics is another story.  As Mr. Johnson puts it in his study, “all districts have been on the same boat continuously since 2003 in mathematics at the 4th grade level.  What means is that the variation shown in the graph is random, and not due to any special cause.

Figure 3. NAEP TUDA, Mathematics, 4th Grade, All Students prepared by edwjohnson@aol.com

There is very little student improvement in reading or mathematics at the 4th grade level as shown in Figures 2 and 3.

As long as we continue to ignore the common causes of variation that exist in the system then we can expect very little to no improvement.

But as Mr. Johnson has said in other letters and reports, if fundamental questions about the purpose of schooling are not addressed and if we can not agree on these purposes, very little will change in the system.  In the two systems explored here, reading at the 4th grade and math at the fourth grade, we need to ask: What is purpose of teaching reading in the elementary school?  Why do we teach reading in the elementary school?  What is goal of teaching mathematics in the elementary school?  Why do we teach mathematics?

As Mr. Johnson has shown, why are these districts on the same boat for the teaching of mathematics?  How can we used systems theory to look at mathematics teaching as a system and answer questions about how to improve mathematics learning?  How can help students develop a love affair with mathematics?

Ed Johnson has examined a lot of data from the standpoint of systems thinking based in part of his work with Edward Deming, and other scholars in the field of systems thinking.

I highly recommend that you check his study which you can get access to as a PDF file here: NAEP TUDA 2002-2011 Views through a Deming Lens.

In the days ahead, I’ll revisit Mr. Johnson’s study, and report on his analysis of the “performance gap” variation that he has depicted as a series of images as shown in Figures 2 and 3.  I’ll also explore systems thinking and school in more detail.