It’s been three weeks since I asked Toby Belfield, principal at Ruthin School, Denbighshire, North Wales if he would be willing to share the evidence upon which he based his assertion that “part of the problem with forcing young people to learn both English and Welsh (arguably, both to a substandard level) is that young people in Wales will continue to be educationally weaker than their peers in England and abroad.” (see this post).
I have not had any response (not even a cursory ‘no comment’) to the two emails and three tweets I sent to him. Perhaps he is too busy to want to get involved further after his public and unreserved apology – it is exams season, after all.
In that apology though, Belfield fails to address his claim about the effects of dual language education. Instead, he engages in special pleading to recast his assertions as something they were not. His original letter stated that dual language education (“forcing young people to learn both English and Welsh”) puts Welsh children at a disadvantage. In it he also contended that the educational system in Wales should have English as its first language. He now says that what he meant was that he rejects Welsh only education, that he thinks finding suitably qualified teachers who speak Welsh would be difficult, and that he supports parental choice. He did not address any of these things in his original letter (indeed, his insisting that schools in Wales should have English as their first language rather contradicts that last point). This special pleading, in lieu of addressing what he actually said in his letter, suggests either that he is unwilling or unable to support his original position or that he is not sufficiently adept at expressing himself in a letter written for public consumption. I will leave readers to consider which of these possibilities should be the greater concern for the parents of children at his school.
But, to the question of evidence. Belfield has shown no interest in addressing the claim he made, so I felt it only right to attempt to do so on his behalf.
First, I want to be clear that I intend only to address the question of the effects of dual language or Welsh Medium education in Wales with substantive, empirical evidence. There are other claims that Belfield makes that could be assessed in the light of evidence. For example, we could look at whether it would be hard to find sufficiently qualified bilingual teachers, whether children with Welsh as their first language do find it harder to get into top universities, whether job prospects are worse for Welsh-speaking school leavers, whether those Welsh-speaking school leavers who want to find employment outside of Wales do find it more difficult, and so on. I will say no more about them, other than to invoke Hitchen’s Razor.
Belfield is quoted as saying that he based his opinion about the precursors of the poor performance of Welsh school children on the NfER report on the PISA assessments of 2012. In 2012 PISA tested the reading, science and mathematics attainment of 15 year-olds in 65 countries. From the UK data, the NfER prepared reports for each of the constituent countries of the union. I obtained a copy of the Wales report referenced by Belfield to see if I could find anything in it that would lend support to the notion that Welsh medium or dual language education is responsible for poor performance among Welsh children.
On reading the report it is easy to sympathise with Belfield’s concern for the Welsh education system. In summary, Wales is said to be performing significantly below the mean average for OECD countries in all subjects assessed, and significantly lower than all other areas of the UK. Notwithstanding that half of the countries in the sample necessarily must fall below the mean average; so far, so concerning.
These data are derived from the averages of all 3305 Welsh students attending a total of 137 Welsh schools that took part in the assessments. So, if you want to draw conclusions about the effect language of instruction has on attainment, the sensible thing to look for is the languages of instruction of the schools that took part in the assessments. Were they Welsh Medium, English Medium, dual language, or all three? If it was all three, is there any breakdown of results by school type that could indicate if any one type is outperforming the others?
In the appendix of the report (page 101) we learn that “The sample in Wales contained Welsh and English medium schools and bilingual schools, although language of instruction was not a stratification variable.” This means that when the 137 schools that took part in the assessments were initially chosen, no attempt was made to sample a representative number of each type of school. So, we know that Welsh and English medium schools and bilingual schools took part in the assessments, but not how many nor in what proportions.
Later in the appendix we are told that a Welsh language version of the test was made available to schools who requested it (page 103). They report that in 14 schools all students completed the Welsh language versions of the tests and that in a further 8 schools some students completed the Welsh language version. The report states that, in total, 381 students carried out the tests in Welsh. From these figures we might infer that 14 of the 137 schools were Welsh medium and that the other 8 schools might have been bilingual schools, or Welsh medium schools with students who nonetheless preferred to take the tests in English.
According to Welsh Government statistics, in 2012 there were 56 (25%) Welsh medium secondary schools, serving 41,262 (21%) children. So if my guesswork is in any way accurate, we might conclude that Welsh medium schools are underrepresented in the PISA sample as a proportion of secondary schools in Wales as a whole (approximately 10% as opposed to 25% – or approximately 16% if we include the schools where only some students took the test in Welsh), as are students taking the test in Welsh, compared to the proportion of children in total who attend Welsh medium schools (11% as opposed to 21%). But, as I say, this is guesswork, and nowhere in the report is this clarified. And nowhere in the report are the results of the assessments broken down by school type anyway. This leaves us none the wiser as to any potential associations between language of instruction and student attainment.
So, what can we conclude from this? Mainly, that Toby Belfield can’t have based his claims about the effects of Welsh language education on anything contained in the PISA report that he suggests informed them.
I wasn’t happy to leave it there though, so I contacted the authors of the report to ask if they would be willing to share the raw data so that I could do my own sub-group analysis based on language of instruction.
Bethan Burge, one of the authors of the report, was extremely helpful and explained to me that the data was owned by the British and Welsh governments and that I would need to ask them if I could see it. However, she did send me a copy of another report that she and a colleague had prepared using the PISA data, entitled Additional Analysis of Wales’ Performance in PISA 2012. In addition to the sub-group analyses you would expect (results by gender, free school meals, ethnicity, SEN and so on) it had a short analysis of the results of the PISA assessments, stratified by medium of instruction.
Here it is in all its glory:
In reading and science, the performance of learners attending Welsh medium and English medium schools is comparable, that is, score differences are not statistically significant. However, in mathematics, learners attending Welsh medium schools outperform those in English medium schools by 10 score points. This difference is statistically significant.
Burge and Lenkiet (2015:7)
In addition there is some analysis of GCSE results and language of instruction:
At the school level, there are significant associations between medium of instruction and mathematics GCSE scores […] if we look at a group of learners with similar scores in the PISA mathematics assessment and similar individual learner characteristics, those in a Welsh-medium school will, on average, have higher mathematics GCSE scores.
Burge and Lenkiet (2015:12)
At the school level, as is the case for mathematics, there are significant associations between medium of instruction and science GCSE scores […] This means that if we look at a group of learners with similar scores in the PISA science assessment and similar individual learner characteristics, those in a Welsh-medium school will, on average, have higher science GCSE scores.
Burge and Lenkiet (2015:12-13)
So, I can now answer my own #AskForEvidence based on the same data source that provoked Belfield to make his bold claim. Far from putting students at a disadvantage, the PISA data suggest that language of instruction makes no difference to outcomes in science and reading (and so guilt-free parental choice can be exercised) and that Welsh medium schools put students at an advantage in maths. In terms of GCSE results, all else being equal, Welsh medium schools are associated with higher achievement in GCSE maths and science.
Of course there are always caveats to any data set, some of which I have addressed in my original post on this matter, and the article linked to by one of the comments on that post is a useful read in this regard. Nonetheless, it is helpful to know that empirical evidence is available to inform assessment of the effects of different models of Welsh education. And empirical evidence is always preferable to personal opinions based, we can only assume, on no more than gut feeling.