Will Georgia follow Florida by releasing teacher VAM-like scores?  In Florida’s case, the Florida Times-Union released links to all 116,723 teachers’ VAM scores in an extraordinary unethical move that has happened other locations, including New York, and California.

The Georgia Department of Education (GADOE) will have two or more sets of data on every teacher in Georgia, and by law (Georgia House Bill 244) has to base 50% of the score on student growth measures (achievement tests).  The drive for a VAM-like system of teacher evaluation is a result of Georgia’s Race to the Top $400 million grant in 2010.  One of the chief components of the RT3 is the building of data systems about student and teacher performance.  As part of the RT3 grant, 26 Georgia districts have collaborated with the GADOE on the grant, and especially a teacher evaluation system.

Here is how it will work in Georgia.  Each teacher will have a summative score generated from three data sources (Figure 1) including assessment of performance based on classroom observations, assessment of teaching based on student surveys, and assessment of student growth using a Value Added Model (VAM).   This system is known as the Teacher Keys Effectiveness System (TKES).  There is a parallel system for administrators (LKES), although it is not discussed here.

Keep in mind that at the end of the day, each teacher will have a score computed by some form of mathematical formula based on these three subsystems, classroom performance, student surveys, and student growth.  It will be out there ready to be published for all to see.

Figure 1. Georgia Teacher Effectiveness System. From the Georgia Department of Education Office of School Improvement
Teacher and Leader Keys Effectiveness Division

Let’s take a look at the three-parts of Georgia’s teacher evaluation system.

Teacher Assessment on Performance Standards (TAPS)

TAPS is the first subsystem, and it will be based on classroom observations and “documentation.”

Each teacher will be observed in the classroom and evaluated based on the Teacher Assessment on Performance Standards (TAPS).  “Trained” observers will visit (announced or unannounced) four times by short walk-abouts, and two full observations of 30 minutes each, for a total of about 100 minutes. Figure 2 is a list of 10 classroom standards, each of which is judged by a trained observer.

Your TAPS score will be based on the ratings on each performance standard listed in Figure 2.  The “trained observer” will use performance appraisal rubrics to rate you.

But here is the deal.  There are only ten performance standards in Figure 2.  But you will be rated on at least five performance indicators for each standard.  For example, according to the GADOE documentation on TKES, seven indicators are used to rate your professional knowledge.  My count based on the TKES document is that there are at least 69 performance indicators in the observational system.  Each of these performance indicators is a behavior that you must show consistently (regular intervals) or continually (frequently–every day, every class).  I did not make this up.  It is in the TKES documentation.

So, according to the state, teaching can be broken into about 69 behavioral performances which are grouped into ten categories or standards.  The GADOE puts it this way:

A Summative Performance Evaluation shall be completed for each teacher which establishes a final rating on all ten Performance Standards. These ratings shall take into account ALL data sources available including student perception data generated by the Surveys of Instructional Practice.

Ratings of Level IV, Level III, Level II, or Level I shall be provided for each of the ten performance standards using the performance appraisal rubrics. The evaluator will rate each of the ten Performance Standards based on the totality of evidence and consistency of practice.  (TKES Implementation Handbook, Georgia Department of Education, Office of School Improvement)

Figure 2. TAPS Performance Standards, Georgia Department of Education.  Each standard will be rated using multiple indicators, and a final score will be generated at the end of the year.

Figure 3. You must either consistently or continually do or show the performance indicators to get a passing rank in the TKES system.

Student Surveys of Instructional Practice

The second subsystem will ask for student opinions of their teachers in Grades 3 – 5, Grades 6-8, Grades 9-12.  This will give the state student perception data on each teacher, Grades 3 – 12.  The data will be used as part of teacher classroom performance.

The claim here is that student perception surveys should be “aligned” (as educators, we like this term) with the performance standards.  The student data will be uploaded to the GADOE Electronic Platform or similar data system.  Another set of data on teachers for the local newspapers to go after.

One section of the documentation that is a bit disturbing here is that student survey data must be included for standards 3, 4, 7, and 8, and if the TAPS ratings and student survey results are inconsistent, then the evaluator must give justification for the difference.  The state is assuming there is a direct correlation between student perceptions and teacher performance.  Where is the evidence?

Here are a couple of sample student survey questions:

• My teacher encourages me to participate in class, rather than just sitting and listening.
• My teacher encourages me to ask questions in class.

Figure 3. Survey results by mean for the four key standards (3, 4, 7, 8).

According to the state, if student survey data differs very much from the mean scores for these standards (Figure 3), then justification must be provided why there is a discrepancy.

This appears to be a nightmare.

VAM: Student Growth and Academic Performance

The State Department uses “student growth percentiles (SPGs) as a measure of student growth.  Don’t be fooled here.  This is no different from VAM.  It is VAM.

Here is what the state says about student growth percentiles:

Student Growth Percentiles (SGPs) shall be used as the student growth component of the Teacher Effectiveness Measure (TEM) for teachers of tested subjects. SGPs describe a student’s growth relative to his/her academically similar peers – other students with a similar prior achievement (i.e., those with similar history of scores). A growth percentile can range from 1 to 99. Lower percentiles indicate lower academic growth and higher percentiles indicate higher academic growth. From the 1st to the 99th percentile, growth is possible for all students regardless of previous achievement scores. Annual calculations of student growth for tested courses are based on state assessment data (grades 4-8 CRCT and high school EOCT). (TKES Implementation Handbook, Georgia Department of Education, Office of School Improvement)

But you need to realize that none of this is based on sound science.  As Diane Ravitch says, this is junk science, and of course I agree.  Here is what the state does.  It takes two years of earlier test data, and calls this data a pretest.  So if you teach fifth grade math, then the pretest data for YOUR CURRENT students are generated using these students 3rd and 4th grade CRCT results.  It is this data against which you will be evaluated as adding value to the student performance in your 5th grade math class.

But hold on.  Get this.  The SPG model will give the state a wealth of student, classroom, school, district, and state growth information based on Criterion-Referenced Competency Tests (CRCT) and End of Course Tests (EOCT).

But, what if you teach a course or subject that is not tested using a CRCT or EOCT.  Because there is no pre-test data available, then there is really no problem here.  Teachers or groups can make up a pre-test, which will be given at the beginning of the course, and then a post-test will be given at the end of the course.  Student gains can then be determined.

The problem is that this data is unreliable and of questionable validity.  The Pre-Post Test model is a valid model of research if there is a control group available for comparison.  But there isn’t.  Any gains or losses in scores can be due to history, maturation, characteristics of students enrolled in your class, changes in the sample of students in your class over the year, effects of pre-testing, reliability and validity of the test, and so forth.

50 – 50

You probably know that our state legislature has mandated that the student growth percentile be counted for 50% of teachers’ evaluation.  The Teacher Assessment on Performance Standards (TAPS) is weighted 50%.  Combining these scores is your evaluation. Teachers will receive a score and a rating level called Exemplary, Proficient, Needs Development, or Ineffective.

Mind you, there is no scientific basis for establishing scores for these four rating levels.  But the state will tell you that they do have the data.

Teacher evaluation is NOT nonsense.  As teachers, we welcome assessment, especially if can enhance our professional abilities and professionalism.  But this system, as rigorous as it is displayed in the state’s documentation is no different from the VAM nonsense that we saw unveiled by the Florida Times-Union.

I predict that the Atlanta Journal-Constitution (AJC) will seek and get all of this data, and publish in the AJC newspaper.

What do you think?  Will Georgia teachers be put on display as were all of Florida’s teachers?