(When) Is it OK to deny half of your pupils a promising new teaching approach?

One of the objections sometimes voiced about conducting RCTs in education is that they are unethical. This, it is often claimed, is because requiring a control group (a group of children who do not receive the teaching approach that you wish to evaluate) wilfully ‘denies’ that group access to the new teaching approach. I encountered this argument at a seminar session at BERA 2016, and explored my response to it this post. I’ve been thinking about it some more.

First, it is important to stress that there is nothing peculiar to RCTs about this allegedly unethical behaviour. Research of any design that evaluates the effects of a new teaching approach by comparing it to an alternative involves ‘denying’ some children access to the new approach. Matched pairs designs, regression discontinuity designs, non-random comparisons, multiple baseline interrupted time series, stepped wedge designs, and designs with a waitlist control, all ‘deny’ the new approach to some children, either for the duration of the evaluation or for a portion of it. Equally, all designs (experiments, quasi-experiments, and non-experimental observations) ‘deny’ access to the new approach to any children who are not in the study. Moreover, and by the same logic, children receiving the experimental approach in any of the above contexts are ‘denied’ the control approach for the duration of the study.

This last point brings us to my second, more germane, observation. Implicit in the argument that children in a control group are ‘denied’ the new approach is the assumption that new approaches are always superior to existing approaches. The argument fails to acknowledge the possibility that a new approach might be inferior to existing approaches. It fails, therefore, to acknowledge that children may be harmed by exposure to a new teaching approach. The argument that control groups are unethical is, therefore, lopsided. By the standard applied here, we must acknowledge that ‘denying’ children access to the control approach is unethical as well, as exposure to the new approach may be harmful. This does not leave us in a very informed position.

A recent tweet by Vinay Prasad (see below) explored this argument in relation to the use of sham surgical procedures in evaluations of new surgical approaches. Here, a new surgical procedure is compared to a surgical placebo. Basically, the members of the control group undergo a surgical procedure (anaesthesia, incision, stitching, etc.) but do not undergo the actual surgery. This is important because the potential harms of undergoing surgery may outweigh potential benefits of the surgery itself. For example, in the treatment of prostate cancer, surgery to remove the cancer can cause complications such as incontinence and impotence without changing the life expectancy of the patient (many men die with prostate cancer, not because of it). Without a sham procedure we are less well informed about the relative quality of life following the surgery and, therefore, we are in a less well-informed position to decide the best course of action for the patient.

Prasad illustrated this in the flow diagram below, making the point that introducing a new surgical procedure without comparing it to an alternative (sham) procedure risks harming patients for no apparent reason.

Harms in education are less easy to spot. They are rarely as dramatic as some of those found in medicine, but they do exist (see, for example, evaluations of Chatterbooks and Mate-Tricks).

It is perhaps better to think in terms of opportunity costs. For example, a child taught using an approach that is less effective than available alternatives may still make progress, but at a slower rate. Or, if a child spends time away from their mainstream classroom to receive a targeted intervention, they miss whatever is going on in their classroom during that time. Or, in cases where a new approach is no better and no worse than existing approaches, there are costs of time, money and effort associated with changing the way teachers teach. What could be done with that time, money and effort instead of implementing the new approach for no relative gain in primary outcomes?

Building on Prasad’s work, I mapped out my thoughts on the use of control groups in education. See below.

 

Ethics of Control Groups in Edcuation

For those who argue that ‘denying’ children access to a new teaching approach is unethical, I invite them to consider the question at the bottom of that diagram. My own position is that when there is uncertainty about the effects of a new teaching approach, the only ethical course of action is to evaluate it in relation to the best available alternative, which necessitates having a control group.

A final note on RCTs. As I have said, there is nothing peculiar to RCTs about ‘denying’ an approach to some children while making it available to others. What is peculiar to RCTs is the method by which children are allocated to different approaches. The single defining feature of an RCT is that children are allocated to alternatives fairly. No child (or school or classroom if you are doing a clustered RCT) stands a better or worse chance of being allocated to either the experimental or control group than any other child when the decision is a random one. That’s the whole point of randomisation.

When my brother and I were children, our dad used to toss a coin to decide which of us got lumbered with the washing up after family meals. While it never felt fair to the one who ended up wearing the Marigolds, we could hardly argue with the ethics of our dad’s method of choosing. By the same token, when we must decide who receives what in a comparison of alternative teaching approaches, I contend that random allocation is not just the most effective way of creating unbiased comparison groups, but it is the most ethical way, too.

For anyone interested, you can download a PDF version of my diagram here. Feedback welcome.

Advertisements

3 comments

  1. Michael

    Does anyone actually argue that control groups are unethical? Opposed to it in principle, not just feeling a bit unhappy about it.

    If teachers’ experience and insight convinces them that a new/different approach is better, they will inevitably regret it if their students aren’t benefitting. Doesn’t mean that they reject evidence-based research – simply that they are doing their job, to fight for the wellbeing of THEIR students.

    Although I suppose this article is addressing both (and doing so well), so fair enough.

    • Hamish Chalmers

      Thanks for your comment Michael.

      In answer to your first question: Yes, people do. However, I think that even those most ideologically opposed to ‘denying half of your children a promising new approach’ would concede that in order to find out whether a new approach is better, worse or no different to existing alternatives some sort of comparison must be made. The question then is what they think is the least biased and most ethical way of generating that comparison. My view is that, where possible, random allocation to alternatives is the least biased and most ethical. If others feel different they must explain what the alternative should be and why they think it is less biased and more ethical than random allocation to alternatives.

      As to your second point: I agree that teachers are advocates for their students and want to do the best by them. Experience and insight are important factors in evidence based education (if you’re interested, I wrote about the intersection between external research evidence, teacher expertise and contextual factors as a triumvirate for Evidence Based Education here http://www.cebm.net/blog/2016/07/06/can-education-learn-evidence-based-medicine/). However, with the best will in the world we can be mistaken. You are right to say that a teacher who is convinced that they are helping their students with a new approach who then tests that approach against an alternative and finds that they were right all along may regret the period during which half of the children received the inferior approach. However, the flip side is when they are convinced but *wrong*. There is no shame in conceding that our best intuitions are sometimes woefully incorrect. In the absence of any external research evidence that would address uncertainty, a teacher who refuses to test their convictions in an unbiased comparison because they *know* they are right is acting unethically. If they are wrong about their convictions about a teaching approach then every student they ever teach using that approach will be denied a superior approach, rather than just the group who participated in the trial.

      I guess that you already understand that given your final comment, which is great.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s