Jeremy Corbyn used all six of his questions at PMQs on Wednesday 20th April to take David Cameron to task over the planned forced academisation of schools. It was a good debate and Corbyn was dogged in his persual of answers to the question of why the government is taking this approach against the wishes of much of the profession and many of Cameron’s own MPs.
It’s worth watching the whole exchange, and you can do so here.
In the course of it, it struck me that Cameron could do with some advice on basic logical fallacies and statistics. So in that spirit I have isolate two mistakes Dave made that he and we should be aware of.
First is base-rate fallacy.
Converter Academies initially were those that were already ‘outstanding’. These were pre approved for conversion, a dispensation that was quickly afforded to ‘good’ schools as well. A House of Commons briefing paper on academies puts it like this:
“[Pre-approval] has been extended to all schools that are deemed as ‘performing well’” p3
If Cameron is suggesting that it was the act of conversion (and what followed) that led these schools to become “either good or outstanding” then he is fooling himself, or trying to fool us.
That same HoC briefing paper goes on to note that:
“Analysis of 2013 exam results appears to show more progress amongst converter academies than all non-academy schools, especially among the very first converters, that became academies in 2009/10. These schools were all rated ‘outstanding’ by Ofsted at the time, so greater progress made in 2013 might be better explained by pre-existing differences rather than the impact of academy status.” p7-8
Moreover, if only schools that were good or outstanding were allowed to convert, what are we to make of the 12% of converter academies that are not now considered good or outstanding?
Which brings us to the second of Dave’s misunderstandings: regression to the mean
Any variable that is extreme on first measurement will tend toward the average (Mean) on its second measurement. Daniel Kahneman describes this statistical phenomenon very well in is book Thinking Fast and Slow. He does so especially clearly in his report of a flight instructor who noticed that every time he bollocked a trainee fighter pilot for an extremely bad manoeuvre, the trainee subsequently improved. By contrast, every time he praised one for an extremely good manoeuvre the trainee subsequently got worse. This instructor concluded that praise was useless and bollocking was an important pedagogical tool. In fact what he was witnessing was regression to the Mean (and nothing to do with the effects of his approach to critical feedback). Extremely good manoeuvres are extremely rare, as are extremely bad ones. The only place to go from an extreme position is back towards average. These pilots were very unlikely to maintain their either awesome or appalling behaviour in the air and would naturally tend back towards more average behaviour the next time they went out in their planes. This made some look worse and others look better.
This phenomenon is just as true for schools. Ignore for the moment the woefully inadequate judge of a school effectiveness that we call Ofsted, and think about the likelihood that a school will be in ‘special measures’. According to the 2013/14 report from the Chief Inspector of Schools, only 3% of schools across all state maintained sectors were judged to be inadequate. This is an extreme value. Any school that is judged as inadequate has only one direction to move in. Inevitably it will move in that direction, regardless of whether it is in the hands of an academy or not.
Perhaps if Dave looked at the other figures for inadequate schools in that document he might be more inclined to learn about statistical analysis of data. As I have said, the national average for schools judged inadequate over all sectors is 3%. The proportion of inadequate academies (taking converter and sponsor-led together) is 10% of primary school academies, 17% of secondary school academies, 5% of pupil referral unit academies, and a whopping 42% of special school academies.
Should Corbyn point out Cameron’s misunderstandings of basic logic and stats next time this discussion comes up? I think he should.