In March 2014, Jeb Bush’s organization Digital Learning Now (DLN), issued its 2013 Digital Learning Report Card measuring and grading K-12 education policies in the nation’s 50 states against its 10 Elements of High Quality Digital Learning.

I found their report misleading and disingenuous.

Digital Learning Now released its report card grading each state on 41 criteria divided into 10 Elements of High Quality Digital Learning.  You can visit their website to find interactive maps and a full report.

The ten elements are policy statements that the FEE claims are the essential elements for a high quality digital learning environment.  The 10 Elements are shown in Box 1, along with one of the criteria that states must adhere to or be marked down.

According to the Digital Learning Now website, 41 criteria categorized into the 10 elements for their rubric which according to them, “allowed for an objective evaluation of policies across all states.  Using research-type language, they weight equally each of the 10 elements by grading each criteria (41 of them) on a 0 – 4 point scale.  Thus scores can range from 0 – 164.

Each state completed ONE survey and returned it to DLN for analysis, and follow-up, if needed.  According to the Bush group, staff consulted with several groups, none of which were universities or schools, but all were either private firms, or those with a financial interest in virtual schools and digital curriculum.  The Bush digital foundation would have us believe that have a survey instrument which can be used to check the state of state’s digital policies. They use terms such as metric, which when you see the criteria you will at once notice that most of the “criteria” are based on Jeb Bush’s “Florida Miracle.”

In her new book, A Chronicle of Echoes (Library Copy), Dr. Mercedes Schneider highlights the Bush plan (in three chapters) for corporate education reform.  Dr. Schneider shows who Bush, through several Foundations is using his model for self promotion:

One could consider Bush’s statement, that Florida education reforms are “now a model for the nation,” from two different perspectives.  First, one might view such a statement to mean the Florida education reforms actually work, and are “a model to the nation.”  Second, one might consider that, regardless of the efficacy of these Florida reforms, model legislation has been written and is being actively marketed to states across the nation as the panacea to “reform” education.  Bush himself promotes both views.

Digital Learning Now is a way for Bush to package his “reforms” but in the context of digital learning and virtual schools.  Schneider identifies the following as the six key parts of the Bush education reform plan:

1. Grading schools on a A through F scale based upon student standardized scores.
2. Using of high-stakes testing.
3. Preventing student social promotion.
4. Basing teacher pay upon student performance on standardized tests.
5. Using nontraditional avenues for teacher credentialing.
6. Supporting charter schools, vouchers for private schools, and online schools (“parent choice”)

These are all present in the Digital Learning Now plan, and in its survey instrument.

Box 2 shows three criteria which are used to assess the Eligibility, one of the 10 elements of high quality digital leaning.  Note the word “must” in the first two criteria, and note that criteria #2 says that the state must require every student to take at least one online course to graduate. Who will benefit from this criteria? We see here authoritarian tactics used to promote a political and corporate plan in a democratic society.

So, one question to ask here is, How did the states do on “eligibility?”   Thirty states got a grade of “F,” 15 got a grade of “D,” and only 5 passed.  And by-the-way, Florida was rated highest, getting a 100% on this element.

You can see the results at the Digital Learning Now website.  Using a series of maps, you can click on an element and see at a glance how the country did as a whole, or zoom in on a state and see its grade.  Figure 1 are the grades for each state based on their overall score.  Notice that only two states got an A, a few Bs scattered here and there, a lot of Cs in the midwest, but Ds and Fs elsewhere.

Figure 1. Overall Grades on the Digital Learning Now Score Card, 2013. Source: http://digitallearningnow.com/

Misleading and Disingenuous

The criteria that the Bush Foundation has identified to rate the states is designed to support their political views, and financial assets.  The Digital Learning Now group is nothing more than a politico-digital-wing of the Foundation for Excellence in Education.

None of the data that they have collected would be acceptable if they tried to publish an article using the methods, tactics, and so-called “metrics” of their report.

The Bush group converts the scores they obtained from one questionnaire per state into a grade.  Not only does this lack condor, it misrepresents what the states are doing in digital learning.  For example, as I’ve stated, the largest score on the questionnaire is 164.  But the Bush group does not use real scores.  Instead they convert them to percentages, and then using a conversion chart of their making, they give each state a grade as follows:

Figure 2. Grading scale used on Bush’s Digital Learning Now Report Card. Source: http://digitallearningnow.com/

There is no scientific basis to this conversion scale. The cut offs are opinion on qualitative and personal viewsof the Digital Learning Now staff. Nothing more. Nothing less.  There is no basis for deciding that a score lower than 59% is an F, any more than a score above 90% is an A.

In their report 27 states were graded “D” or “F.”  Or to put it another way, 54% of the states seem to be digitally challenged.  To to make matters worse, another 22% were graded “C,” meaning less than a fourth of the states digitally qualified.

What if the data was analysed in a different way?  What follows is an analysis of the Bush data using somedescriptive statistics and a more robust statistical process control.   If the Bush team did this, their report would read very differently.  But remember, if the Bush Foundation can show how poorly states are doing, then they put themselves into a position of pushing their reforms onto the backs of citizens in other states.  There is a lot of money to be made in the digital world, and if you study the Bush Foundation rosters, you will see that its stacked with people ready to make the move.

I converted all the percentages to real scores earned by each state.  Then, I examined the data using these raw scores.

The mean score on the questionnaire was 111 and median score was 118, and the standard deviation was 19.2.  The scores ranged from 67 – 151.

Figure 3 is a histogram of scores which shows a nearly normal distribution for how the states scored on the DLN score card.  It’s a normal distribution.

Figure 3. Histogram of Scores for States 2013

Variation

If we consider the variation in the scores, we find something very interesting about digital learning as measured by Bush and his team. Take a look at Figure 4. This is a flow chart of the scores that were released by Digital Learning Now.

Figure 4. 2013 Digital Score Card for All States

There is variation from one state to another, but the variation is within Upper and Lower Control Limits.  No state (even Florida) fall outside the control limits.  The Bush report card is disingenuous because it fails to acknowledge that all states fall within expected limits, and that there is no state that needs to be “turned around,” or all of a sudden blamed for failing to meet their standards. Giving states a grade is dishonest.  Indeed, Figure 4 shows that all the states fall within expected limits using Bush metrics!

Organizations such as the Bush Foundation use tactics that are on the edge of being unethical, if not unscientific.  They use “instruments” to collect data from a few people, and then use these results to make outrageous claims about the state of education.  How can 50 questionnaires be representative of the nation?  Come on.

Do you think Bush’s Digital Learning Report Card is Misleading and Disingenuous?