NCTQ Report on Teacher Prep: the Devil is in the Detail.

I decided to read the National Council on Teacher Quality (NCTQ) Report on Teacher Prep to try to learn what the NCTQ had to say about teacher prep in the U.S.

Last week, the National Council on Teacher Quality released its report on Teacher Prep.  Since its release, there has been an explosion of articles and blog posts written about the NCTQ report.  If you Google “NCTQ report teacher prep” you’ll get about 51,500 results.  I didn’t look at all the results, but I did survey the first two pages and I found these results:

• 16 articles were critical of the report
• 2 articles were supportive
• 1 article was neutral
• 2 articles were links to the NCTQ report

Of the first 21 articles, 76% were critical of the NCTQ report.  Authors of these articles, which included bloggers, Dean’s of Colleges of Education, professors, and professional associations, questioned the method used.  In fact, Linda Darling-Hammond reported that because of concerns with the methodology most schools of education refused to take part in the study.  Some writers suggested that the NCTQ study was coercive, and did not ask colleges to take part or become partners in the study, but instead resorted to legal means, and the Freedom of Information Act to get their data.  But others spoke out against the partisan nature of the NCTQ which is funded by conservative groups, and according to these writers, the NCTQ is only pursuing an agenda of putting traditional teacher prep out of business, and replace it with alternative certification programs such as Teach for America.

Disclaimer: I am professor emeritus of science education, Georgia State University (GSU). I was a professor of science teacher education at GSU from 1969 – 2002, coordinator of science education and co-developer of alternative, undergraduate, and graduate teacher prep programs. I also was visiting professor in teacher education programs at the University of Vermont and the University Hawaii, Hilo. I taught science teacher education seminars for more than 20,000 teachers in the Bureau of Education & Research.
It’s a consumer report of a large number of teacher preparation programs which NCTQ claim prospective clients can use to choose a teacher prep program to attend.  However, as Linda Darling-Hammond says, the report is nonsense.

It’s a 112 page report that include colorful graphs, charts, tables and descriptive statistics. The authors? I’m not sure, but there are lots of names, hundreds of corporate and foundation sponsors, but in the end no distinct or verifiable authors. There is also no evidence that the “study”  was reviewed by respected scholars in the field of education research.

There is no review of the literature on teacher preparation in the NCTQ report.  There are some references that you have to search for in the “Notes” section at the end of the report.  These “studies” were either done by the NCTQ, or are studies they cherry picked from the literature to support their political views.

Investigating the Enemy

When you read the NCTQ report it seems as if teacher prep institutions are the enemy. For more than thirty years I’ve read and studied educational research articles published in refereed journals such as the Journal of Research in Science Teaching, Science Education, as well many Handbooks of research in science education, teacher education and the learning science. In these instances, I’ve never read a study in which researchers demanded cooperation from the research participants. The NCTQ policy is very clear. If you don’t give us what we want well use legal means to get it. They also “reached out” to a few students to supply materials that were requested from the administration.

The so-called NCTQ researchers not only resort to coercive strategies to get data (syllabi, curriculum, etc.), but you get the feeling that they snoop around universities, trying to find what texts are used by bookstore shopping.

Measuring Everything Under the Sun

But here is the thing. If you look at Figure 1 (Figure 40 in the NCTQ Report), the data sources for the 17 criteria used to evaluate teacher prep institutions is spread out and far-ranging.  For each criteria there are sources within and outside the higher education institutions, but you have no idea what value is attached to each, and its kind of murky when you begin looking at each source of data.  Take course syllabi.  In many instances, NCTQ sifters had trouble getting syllabi.  Some universities refused to send them, so the NCTQ resorted to their lawyers to impose legal justification to try to get the sources of data.  Eventually they resorted to the Freedom of Information Act.

What we see here is a contentious relationship between the NCTQ and the nation’s teacher preparation institutions.  What kind of results will emerge with the aggressive nature of this “study.”

Figure 1. This is Figure 40 from the NCTQ report on teacher prep. I’ve annotated the report to point out some of its limitations. Extracted on 6/21/2013 from http://www.nctq.org/dmsView/Teacher_Prep_Review_2013_Report

NCTQ has put together a mass of data, trying to measure everything under the sun.  Yet, the kind of data that they are collecting really doesn’t tell us much about teacher preparation per se.

Figure 1 is a figure in the NCTQ report which is an analysis of the criteria used by NCTQ to assess teacher prep programs.  All of the data come from paper or online documents.  None involved interviews or discussions with people at the teacher prep institutions.  As hard as this is believe, it is the pattern that the NCTQ has followed since it was formed by the Thomas Fordham Institute.

In Figure 1, look at the left hand column of IHE’s (data sources from the institutions).  Listed are six sources of data:

1. Syllabi
2. Required textbooks
3. IHE catalogs
4. Student teaching handbooks
5. Student teaching evaluation forms
6. Capstone project guidelines (including teacher performance assessments)

Trained analysts then check 17 standards by using a scoring system after “a very methodical and systematic process of coding and sorting.  Analysts have been trained to follow a very detailed and systematic standard-specific protocol to make a “yes” or “no” decision about whether each of the standard’s indicators is satisfied.”

But there is so much data for each standard its not believable that any kind of reliable or valid system emerges from this “corporate spray.”  The idea is to throw as much at the wall as possible and look for what sticks.  In this case, not much.

Trophies and Stars

NCTQ Gold Trophy for Strong Design

Nevertheless, NCTQ charges ahead and rates institutions by standard.  If an institution meets the standard (according to NCTQ), they are awarded four stars.  Nearly meet the standard= three stars; partly meet the standard and you get two stars; meet a teeny tiny part, one star.  No stars if the institution doesn’t meet NCTQ’s standard.

The “gold trophy” is awarded on some criteria to those institutions with a “strong design.”  And they get five stars!

Because there are so many sources of data, and because many institutions simply did not want to cooperate with NCTQ, there are serious questions about the results.

For example to check how institutions selected students for their programs, there is no way of knowing the relative importance of the data collected.  This is true for nearly all of the standards used by NCTQ.

Evaluating Student Teaching: You’ve Got to be Kidding

Junk Thought, Junk Science

Susan Jacoby, in her book The Age of American Unreason, helps us understand the quite pervasive phenomenon in which anti-rationalism and contempt for countervailing facts and expert opinions manifest itself as the truth.  Jacoby point out that junk thought can come from the right as well as the left.  Accusing each other of irrationality thrives.  But, she suggests that junk thinkers see “evidence as a tiresome stumbling block to deeper, instinctive ways of knowing.” (Jacoby, Susan (2008-02-12). The Age of American Unreason (Kindle Location 3798). Knopf Doubleday Publishing Group. Kindle Edition).

The NCTQ report on teacher preparation is junk science.  The method that they employed in their study avoided data from the very sources that could help uncover the nature of teacher preparation.  These sources are faculty, administrators, students, and cooperating school districts and educators.  Without interviewing and observing teacher preparation programs directly, and without establishing a cooperative relationship with the these institutions, the NCTQ condemns itself to false claims, outright opinions that have little bearing on the nature of teacher preparation.

The conclusions NCTQ makes has nothing to do with the data they collected. Their conclusion is a political statement

The NCTQ is averse to evidence and scientific reasoning. Instead of reviewing the literature on teacher education programs, the nature of these programs, and what makes for effective teacher prep, the NCTQ starts with the premise that teacher preparation in the U.S. is a failure. They cherry pick studies in the literature (only a very few) that support their distorted picture of teacher prep, then use unscientific and nontransparent methods that are impossible to replicate. Honestly, I can’t figure out how they arrived at their rankings.

That is, until I read the two-page conclusion near the end of the report.

According to NCTQ, teacher prep institutions do not “arm” novice teachers with practical tools to succeed in the classroom. Mind you, the NCTQ study did not collect any data about “tools” that were or were not in the teacher prep curriculum, nor did the survey students in any program, or conduct site visits to see teacher educators at work.

Another conclusion NCTQ makes is that teacher prep programs make candidates show their feelings and attitudes about race, class, language and culture through in-class dialogue and journal writing. They never visited classrooms to observe these dialogues, nor did they read any student journals.  Again, no where in the report is there any data related to this politically charged conclusion.

None of the remarkable conclusions are related to the “data” NCTQ collected.

One more thing

Walsh and her colleagues really seem to have a disdain for teacher education. They believe that it is the job of teacher educators to train candidates for teaching much like the Teach For America (TFA) program does in its 5 week teacher prep program. NCTQ reels when teacher educators suggest that their mission is to prepare candidates, and not train them.

The NCTQ is a wonderful example of not only junk thought, but is the epitome of junk science.

What is your opinion on the NCTQ report on teacher prep?