Key Concepts for assessing claims about treatment effects and making well-informed treatment choices

Background

You cannot make informed decisions without information. For decisions about actions to improve or maintain the health of individuals or communities (‘treatments’) to be well-informed and not misinformed, you need reliable information about the effects of treatments. Unfortunately, we are bombarded with claims about the benefits and harms of treatments, many of which are not reliable. Therefore people need to learn how to distinguish reliable from unreliable claims.

Unreliable claims about the benefits and harms of treatments are made in the mass media and social media, as well as in personal communications with family, friends, quacks, and health professionals^1–10. They are made by governments, celebrities, journalists, advertisers, researchers, gurus, aunts, and uncles. They include claims about medicines, surgery and other types of “modern medicine”; lifestyle changes, such as changes in what you eat or how you exercise; herbal remedies and other types of “traditional” or “alternative medicine”; public health and environmental interventions; and changes in how healthcare is financed, delivered, and governed.

Many, if not most people are unable to assess the reliability of these claims. For example, in a survey of a random sample of Norwegian adults, we found that less than 20% of respondents recognized that lung cancer can be associated with drinking alcohol but not necessarily caused by it¹¹. This mirrors misleading claims that are commonly made in the media. For instance, stories about coffee frequently use language suggesting that cause and effect has been established, such as “coffee can kill you”, when reporting on associations that have been found between drinking coffee and various health outcomes¹². Personal experiences (anecdotes) are often used as a basis for treatment claims, and people are more likely to trust anecdotes than research. For example, surveys in the UK have shown that only about one third (37%) of the public trust evidence from medical research, while about two thirds (65%) trust the experiences of friends and family¹³. In addition, anecdotes often exaggerate the alleged benefits of treatments (for cancer, for example) and ignore or downplay harms¹⁴. At the same time, people in need or desperation hope that treatments will work and ignore potential harms.

Consequences of people’s inability to assess the reliability of treatment claims include overuse of ineffective and sometimes harmful treatments and underuse of effective treatments, both of which result in unnecessary suffering and waste^15,16. For example, billions of dollars are wasted on alternative medicine and nutritional supplements for which there is no reliable evidence of benefits^17,18. At the same time, millions of children die unnecessarily, in part because their parents do not seek and use effective treatments that are available to them^19,20, and they don’t trust reliable claims about effective preventive treatments such as effective vaccines²¹.

To address this problem, the Informed Health Choices (IHC) group is developing and evaluating resources to help people learn how to assess the trustworthiness of treatment claims and make well-informed decisions about treatments^22,23. The first step in this work was to identify the key concepts that people need to understand and apply to do this^24,25. We refer to these as the IHC Key Concepts. We review and update this list of concepts yearly. In this article we report the changes that we have made to the IHC Key Concepts since they were first published²⁴ and present the most recent (2018) version.

Methods

The IHC Key Concepts are standards for judgment, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (research) used to support claims, and for making treatment choices. The list is intended to be relevant to people everywhere and to any type of treatment. Many of the concepts can be learned and used successfully by primary school children^22,26,27. Although we have developed and framed the Key Concepts to address treatment claims, people in other fields have also found them relevant. Work to adapt these concepts to apply to interventions in other fields is ongoing, including agricultural, economic, educational, environmental, international development, management, nutrition, policing, social welfare, and veterinary interventions.

The IHC Key Concepts are a starting point for developing learning resources to help people make judgements about the trustworthiness of claims about the effects of treatments (and other interventions), and to make well-informed decisions about treatments. They are also the basis for a database of multiple-choice questions that can be used to assess people’s abilities to assess treatment claims and make treatment choices²⁸. We have written the concepts and explanations in plain language. However, some of them may be unfamiliar and difficult to understand. The Key Concepts list is not intended to be a learning resource. It is a framework that can be used by teachers and others to identify and develop learning resources.

To develop the IHC Key Concepts, we first extracted all of the concepts addressed in Testing Treatments²⁹, a book that was written to promote more critical public assessment of claims about the effects of treatments. We then searched the literature for other relevant material, including books and checklists for the public, journalists, and health professionals²⁴. We also considered concepts related to making judgements about the certainty of evidence of the effects of treatments³⁰.

Our aim has been to include all concepts that are important for people to consider. At the same time, we have tried to minimise redundancy. We have organised the concepts in a way that we believe is logical, and we have sought feedback on this logic. The concepts are not organised based on how complex or difficult they are to understand and apply, or in the order in which they should be taught.

We have collected structured written feedback on the Key Concepts using a form with four questions (Box 1). We initially obtained feedback from 29 members of an international advisory group²⁴. We have subsequently obtained responses to these questions at three workshops:

In addition, we have sought feedback and suggestions from colleagues when we have presented the Key Concepts, and on our website. The Key Concepts are updated yearly, and once or twice each year the three authors review and discuss each new suggestion and feedback from workshops, and we reach a consensus on which, if any, changes to make to the Key Concepts. For each suggestion, we record our response and the rationale for it. We invite comments on planned revisions from the IHC group and others prior to finalising each update.

Three other sources of input have contributed to changes that we have made to the IHC Key Concepts. First, experience from developing learning resources and teaching has led to changes. For example, development of primary school resources³¹ led to reorganising the concepts into three groups from the original six groups²⁴.

Second, we are reviewing related frameworks for critical thinking³², including frameworks for teaching and learning critical thinking^33–37; scientific reasoning, literacy, and thinking^38–41; epistemic cognition⁴²; causal inference⁴³, problem solving⁴⁴, and meta-cognition⁴⁵; health literacy^46–48; and evidence-informed decision making and evidence-based practice^49–51. In addition to ideas for new concepts, this review has contributed to the development of lists of competences (required skills, knowledge, or capacity to do something) and dispositions (frequent and voluntary habits of thinking and doing) for thinking critically about treatments. We added these to the IHC Key Concept list in 2018.

Third, adaptation of the IHC Key Concepts to claims and decisions about other types of interventions (such as educational, economic, and environmental interventions) has contributed to changes that we have made, including the decision to reorganise the Key Concept list in 2018.

Results

The 2018 version of the IHC Key Concepts is the most recent version. It can be found as Supplementary File 1 and online⁵². Before reporting the changes that we made in this version and the reasons for those changes, we summarise the changes that we made to the IHC Key Concepts in 2016 and 2017.

The first version of the IHC Key Concepts, published in 2015²⁴, included 32 concepts in the following six groups:

In 2016⁵³, we added two new concepts and reorganised the concepts into three groups. The two new concepts were:

The decision to reorganise the concepts into three groups grew out of our efforts to simplify the concepts and teach them to primary school children. The suggestion to use three groups - claims, comparisons, and choices - came from Matt Oxman, who had primary responsibility for writing the text for The Health Choices Book for primary school children⁵⁴. The book, which has been shown to be an effective learning resource in a randomised trial with over 10,000 children in Uganda, is a story in comic book format which introduces and explains 12 Key Concepts.

In 2017⁵⁵, we added short titles for all the concepts and two new concepts:

The suggestion to add the short titles came from Douglas Badenoch, the project manager for the Testing Treatments websites⁵⁴. The short titles were needed for the Critical thinking and Appraisal Resources Library (CARL) on the Testing Treatments - English website. CARL is a database of learning resources for teachers and others who are responsible for encouraging critical thinking about treatment claims⁵⁶. It contains over 500 open-access learning resources in a variety of formats, including text, audio, video, webpages, cartoons, and lesson materials. Each resource is relevant to at least one IHC Key Concept and CARL can be searched or browsed using the Key Concepts.

In the 2018 version (Supplementary File 1), we merged two Key Concepts and added nine new concepts. We reorganised the concepts within each of the three main groups and added three subgroups to each of the three main groups of concepts. We also replaced all of the short titles and introduced emojis.

We removed the concept that “hope or fear can lead to unrealistic expectations about the effects of treatments” and incorporated this in the explanation of the concept “treatments may be harmful”. The explanation begins with “People often exaggerate the benefits of treatments and ignore or downplay potential harms.” We added: “Similarly, people in need or desperation hope that treatments will work and ignore potential harms.”

The nine new concepts were:

We introduced three higher level concepts within each of the three groups of Key Concepts and reframed the titles of the three groups as shown in Box 2.

We did this in response to feedback that the organisation of concepts within the three main groups was not logical, and that having long lists of concepts was overwhelming. The subgroups of concepts, using these higher-level concepts, provides a more transparent logic for how the concepts are organised in each main group. Having just three higher level concepts for each group may also make it easier to get the gist of the concepts and make the list less overwhelming and easier to remember.

There were three reasons for changing the short titles used for each of the Key concepts. First, we had received feedback that the short titles were not consistent with some of the concepts and that some were not short; and it was difficult to come up with a short, catchy title that accurately reflected each concept. Second, we wanted short titles that were consistent with the new organisation of the concepts. Third, short titles that we were developing for posters and a website targeted at school children seemed to be a solution to this problem. We added emojis to make the poster and website that we are developing more appealing. When presenting these to colleagues and others, the emojis appeared to appeal across age groups and to reflect the content accurately, which also may help to convey the gist of the concepts. The full list of short titles for the Key Concepts and the emojis are shown in Box 3.

Box 3. Overview of the 2018 version of the Informed Health Choices (IHC) Key Concepts (short titles)

1. Beware of treatment claims like these We hear claims about the effects of treatments all the time. Many of these are not trustworthy. When you hear someone use one of these reasons to support a claim about the effects of a treatment, you should beware and ask where the evidence is.
1.1 Beware of claims that are too good to be true.     a)  “100% safe!”     b)  “100% effective!”     c)  “100% certain!” 1.2 Beware of claims based on faulty logic.     a)  “Treatment needed!”     b)  “It works like this!”     c)  “Associated with!”     d)  “Real world data!”     e)  “No comparison needed!”     f)  “A study shows!”	g)  “Old is better!”     h)  “New is better!”     i)  “More is better!”     j)  “Early is better!”     k)  “Personalised medicine!”     1.3  Beware of claims based on trust alone.     a)  “As advertised!”     b)  “It worked for me!”     c)  “Recommended by experts!”     d)  “Peer reviewed!”
2. Check the evidence from treatment comparisons A treatment has to be compared to something else to know what would happen without the treatment. For treatment comparisons to be FAIR, the only important difference between comparison groups should be the treatments they receive. Unfair treatment comparisons and unsystematic summaries of treatment comparisons can be misleading. The way that treatment effects are described can also be misleading.
2.1 Don’t be misled by unfair comparisons!     a)  Dissimilar comparison groups     b)  Indirect comparisons     c)  Dissimilar attention and care     d)  Dissimilar expectations or behaviours     e)  Dissimilar assessment of outcomes     f)  Unreliable assessment of outcomes     g)  Lots of people not followed-up     h)  Outcomes counted in the wrong group 2.2 Don’t be misled by unreliable summaries of treatment comparisons!     a)  Unsystematic summaries	a)  Selective reporting     c)  Unfounded assumptions 2.3 Don’t be misled by how treatment effects are described!     a)  Just words     b)  Relative effects     c)  Average effects     d)  Few people or events     ae)  Subgroup analyses     f)  Statistically significant     g)  No confidence interval     h)  No evidence
3. Make well-informed treatment choices Deciding what to do requires judgements about the relevance of the evidence, how important the good and bad outcomes are to you, and how sure you can be about the treatment effects.
3.1 What is the problem and what are the options?     a)  What is your health problem and what are your     options? 3.2    Is the evidence relevant?     a)  What outcomes matter to you?     b)  Are the people (or animals) very different from you?	c)  Are the treatments different from those available to you?     d)  Are the circumstances different from yours? Do the advantages outweigh the disadvantages?     d)  Do the advantages outweigh the disadvantages for you?     d)  How sure are you about the treatment effects?

Discussion

Up to now we have received much positive feedback, along with many suggestions for improvements, on the IHC Key Concepts, including positive feedback on the changes that we made in the 2018 version. Nonetheless, as can be seen from the results reported here, we have made many changes since the Key Concepts were first published in 2015. There are now 44 Key Concepts compared to the original 32; the concepts have been reorganised from six to three groups; we have added higher-level concepts within each of those groups; we have added short titles; and we have made changes to many of the concepts. We will continue to revise the IHC Key Concepts in response to feedback. Although we and others have found the concepts helpful since they were first published²⁴, we anticipate that there will still be ways in which they can be further improved. We welcome suggestions on ways of doing this.

The most common misunderstanding in the feedback we have received is that the Key Concepts list is a learning resource intended for people with no relevant research background. As noted in the Methods section, the list of Key Concepts serves as the basis for developing learning resources. It is not designed as a learning resource. It is a framework, or starting point, for identifying and developing learning resources.

Another common misunderstanding is that the Key Concepts are organised in the order in which they should be taught or learned. We have organised the Key Concepts logically by grouping them first in three groups and then within those three groups using higher-level concepts (Box 2). This logic does not reflect the difficulty of the concepts or the order in which they should be learned.

When teaching the concepts, it may make sense to start with ones in the first group, followed by ones in the second group, followed by ones in the third group. However, it does not necessarily make sense to teach them in that order or in the order that they are organised within each group. For example, at least 24 of the Key Concepts can be understood and applied by primary school children³¹, whereas other concepts are likely too difficult for primary school children to understand and use. Thus, it would obviously make sense to hop over those concepts when teaching primary school children.

Also, it is important not to try to teach or learn too much at one time. We initially tried teaching 24 Key Concepts to primary school children in one go, and found that was too much to teach in a single school term³¹. Our efforts to teach IHC Key Concepts to both primary school children and their parents support our initial hypothesis that the time to start learning these concepts is in primary school - if not even younger⁵⁹. Ideally, these concepts should be taught and learned using a spiral curriculum^60–62, that maps out what students should learn, where they should begin, and how they should progress to master these skills. Each cycle in a spiral curriculum reinforces what was learned previously while introducing new concepts. This can help teachers and students identify when milestones have been reached, build a foundation for later stages of learning, and guide the development of assessment tools and learning resources. We have not yet developed a spiral curriculum based on the IHC Key Concepts.

Decisions about the suggestions we have received have been based on logic and discussion. Four criteria have emerged from these discussions, which we will use explicitly in further developing the IHC Key Concepts. New Key Concepts have to:

In addition to continuing to seek and review feedback and suggestions, we will further develop the Key Concepts by continuing to learn from using the IHC Key Concepts, other relevant frameworks, and adaptation of the IHC Key Concepts to other types of interventions. We also plan to summarise the evidence supporting each of the Key Concepts.

Conclusions

The IHC Key Concepts have proven useful in designing learning resources, evaluating them, and organising them²⁵. The most recent version of the Key Concepts improves on previous versions by incorporating additional Key Concepts, organising the Key Concepts more logically and, we believe, making it easier to get the gist of the Key Concepts. Future improvements will be made based on feedback and suggestions, and ongoing evaluation.

Data availability

Dataset 1: Suggested revisions to the IHC Key Concepts and responses 2016-2018 https://dx.doi.org/10.5256/f1000research.16771.d223532⁶³