Research design

The ‘gold standard’ for testing the effects of any intervention is the randomised controlled trial (RCT), to which all new medicines are subjected. RCTs have an experimental and a control group, and the allocation of participants to the groups is randomised. The experimental group then receives the new drug, while the control group receives a harmless substance (the placebo) packaged to look like the drug. The information about the randomisation is kept encoded until the end of the study, and is hence double-blind: neither the person administering the tablets nor the participants know which is which. Because the effects of personal bias have been removed, this procedure allows you to decide with confidence whether the new drug is effective or not. Indeed, new drugs are very unlikely to be approved unless they have been subjected to RCTs.

Testing training programmes

Tests of the effectiveness of training interventions typically involve participants completing some form of checklist or questionnaire beforehand and then completing the same set of questions afterwards. Any change is then attributed to the training, but such claims are seldom warranted. For one thing, people will almost always indicate a change in their behaviour after training. This is partly because of compliance effects – people saying what they think the researchers would like to hear – but also because of the Hawthorne effect. This was discovered in research into productivity at the Hawthorne plant of the General Electric Company in Illinois, where regardless of what was done productivity seemed to improve, not because of any real change in the workers’ behaviour but because someone was taking an interest in them. These effects can be mitigated to some extent by ensuring a long enough interval between the first and the second testing sessions, but a more important shortcoming in this work is the absence of an appropriate control group.

It is of course very difficult to create a completely controlled test of a training intervention, because the procedure you’re testing is delivered by people and not by tablets or machines. However, simply giving the participants questionnaires before and after the training provides no evidence at all, and with sufficient planning it is possible to approximate to and RCT design. One example of this is the case study of absenteeism using The Challenge of Change.

Sickness-absence:

A controlled study using The Challenge of Change

This study used a large sample of UK Police officers. The sample was large enough (over 140 participants) to allow sufficient statistical power, and they were allocated to experimental and control groups. In medical research all participants would have an equal diagnosis, and to ensure that the two groups of Police officers were comparable across factors such as age range of age and years in service, the allocation to the groups was randomised systematically. The training provided for the experimental groups was The Challenge of Change, while the control groups received training based on the conventional approach to managing stress and resilience.

Unbeknown to the participants, their rates of sickness absence were being monitored. The baseline was the rate for the period before the training, and was based on secondary rather than primary absenteeism. Primary absenteeism is genuinely being incapable of going to work, and might include for example broken limbs. Secondary absenteeism is sometimes referred to as the ‘sickie syndrome’, being well enough to go to work but not wanting to and using sickness as the excuse.

The results of the study were unambiguous: Sickness-absence was significantly reduced in the experimental group compared to the controls over the 11-month period following the training intervention. As a further manipulation, selected subgroups were given follow-up ‘booster’ training, which had additional effects over and above the initial training.