What is evidence anyway?
The idea of rigour is pretty important when it comes to research. In a previous post I’ve expressed the view that we need to acquire and properly understand evidence before investing a lot of money in rolling out interventions. But, that’s not to say that collecting ‘evidence’ for efficacy and effectiveness is always as rigorous as it might appear.
I’m very interested in anti-stigma mental health education, and I do think it’s important to know that approaches to such education are working well before they are rolled out on any large scale. But, how do you know that they work?
Conventionally, quasi-experimental designs are used in which participants complete attitude surveys before and after taking part in an intervention, and differences in attitude scores are compared. Early in my research career, I constructed a questionnaire designed to measure adolescents’ attitudes to mental illness. I followed a rigorous procedure to do this, interviewing young people in order to generate statements to include on the questionnaire, piloting the questionnaire and conducting factor analysis.
Even though my questionnaire was quite well put together, and appeared to be able to produce valid and reliable results, the process of constructing it caused me to develop a highly critical stance in relation to measuring attitudes in this way.
My feeling was that while an analysis of young people’s responses to the questionnaire may able to indicate general trends and patterns in respondents’ views, the instrument was also reductive. In order to generate material for the questionnaire, I held group discussions with young people. The content of these was rich and complex, with individuals frequently holding contradictory views and occupying ambiguous positions in relation to mental illness. The level of detail, nuance and subtlety which I observed during the focus groups could simply not be captured by a questionnaire using a bipolar response scale.
Another limitation of attitude measurement techniques is that they force responses to be recorded as either positive or negative. It became clear through my discussions with young people that ambivalence and confusion were very genuine features of their understanding, which questionnaires are unable to capture.
Not only that, but using a questionnaire to measure attitudes to something as complex as mental illness is problematic for semantic reasons. Most questionnaires about mental illness, including the one I constructed, rely on people responding to the phrase ‘mental illness’, without knowing how the term is understood by individual respondents. Given that ‘mental illness’ can describe such a vast spectrum of experience, from full blown madness, to fairly mild sadness, it may well be that different respondents are thinking about completely different things as they complete the questionnaire.
This has important implications. A common item on attitude to mental illness questionnaires is ‘people with mental illness are dangerous’. If you are imagining a person in the throes of paranoid psychosis, it’s likely that you’ll answer the question quite differently than if you are imagining someone with postnatal depression or anorexia.
not everything that can be counted counts and not everything that counts can be counted
The interpretation of the answer to such a question is also problematic. If someone agrees with the statement, their response is interpreted as indicating a negative attitude to mental illness. However, although most people with mental health problems are not dangerous, some people with mental illness do sometimes do dangerous things. It is possible that someone may agree with the statement on the grounds that they know this, but that they are generally supportive towards people with mental illness and therefore hold a broadly positive attitude.
So, although it’s important to have some evidence which demonstrates that interventions do what they are supposed to do, it’s also important to be critical about the nature of that evidence. As Einstein said, “not everything that can be counted counts and not everything that counts can be counted”.