Skip to main content

Development of the informed health choices resources in four countries to teach primary school children to assess claims about treatment effects: a qualitative study employing a user-centred approach

Abstract

Background

People of all ages are flooded with health claims about treatment effects (benefits and harms of treatments). Many of these are not reliable, and many people lack skills to assess their reliability. Primary school is the ideal time to begin to teach these skills, to lay a foundation for continued learning and enable children to make well-informed health choices, as they grow older. However, these skills are rarely being taught and yet there are no rigorously developed and evaluated resources for teaching these skills.

Objectives

To develop the Informed Health Choices (IHC) resources (for learning and teaching people to assess claims about the effects of treatments) for primary school children and teachers.

Methods

We prototyped, piloted, and user-tested resources in four settings that included Uganda, Kenya, Rwanda, and Norway. We employed a user-centred approach to designing IHC resources which entailed multiple iterative cycles of development (determining content scope, generating ideas, prototyping, testing, analysing and refining) based on continuous close collaboration with teachers and children.

Results

We identified 24 Key Concepts that are important for children to learn. We developed a comic book and a separate exercise book to introduce and explain the Key Concepts to the children, combining lessons with exercises and classroom activities. We developed a teachers’ guide to supplement the resources for children.

Conclusion

By employing a user-centred approach to designing resources to teach primary children to think critically about treatment claims and choices, we developed learning resources that end users experienced as useful, easy to use and well-suited to use in diverse classroom settings.

Peer Review reports

Article summary

Strengths and limitations of this study

Strengths

• We used a user-centered design approach with a multi-disciplinary team.

• We engaged end-users in the entire development process from brainstorming to piloting.

• Non stringent grant conditions permitted ample time to generate and prototype ideas and then iteratively design the resources.

Limitations

• Time constraints in trying to synchronise the design schedule with the already busy school schedule

Summary box

What is already known:

• There is an information overload regarding unsubstantiated claims of benefits and harms of treatments

• People generally lack the skills to assess the reliability of treatment claims

• Lack of resources to teach critical thinking particulary appraising treatment claims in primary schools in both low and high-income countries.

What are the new findings:

• Use of a user-centered design approach to design resources

• Benefits of multi-stake holder collaboration in the design process

How might it impact on clinical practice in the foreseeable future?

• We designed useful, understandable and transferable resources to teach critical thinking that children and teachers found relevant and easy to use in their particular contexts.

Background

People of all ages, in low- and high-income countries, are flooded with both reliable and unreliable information about how to care for their health, including claims about the benefits and harms of treatments (any action intended to improve health) [1]. Unreliable claims come from many sources, including experts, advertisements and family [2]. People’s beliefs in unproven claims about treatments can lead to harm and waste [2]. Although this problem is global, people with fewer resources to spend on unnecessary treatments are disproportionately affected.

Many studies have found that people’s ability to understand and assess health information is often lacking [1, 3,4,5], although there are limitations in how this has been measured [6]. The Informed Health Choices project aims to enable people to assess claims about the effects of treatments, beginning with primary school children.

Why target primary school children?

Research has suggested that children between the ages of 10 and 12 are capable of learning critical appraisal skills [7], and teaching these basic skills is already part of the curricula in some countries [8]. It is possible to reach a large segment of the population before they drop out of school, as many do after primary level in low-income countries [9,10,11]. Finally, teaching children to assess information about treatment effects can lay a foundation for them to make informed health decisions when they grow older, as patients, future health professionals, policymakers and citizens.

A recent overview of six systematic reviews on education interventions in under resourced countries included 227 studies in total, but none of these studies addressed health or science literacy, or critical thinking more broadly [12]. Systematic reviews of teaching children critical appraisal skills in health also have not found studies of strategies for teaching these skills to primary school children in both low and high income settings [6, 13].

We developed the Informed Health Choices (IHC) primary school resources to help children begin to learn critical appraisal skills required to assess benefits and harms of treatments. Our objective was to design resources that children and teachers experienced as useful, easy to use, understandable, credible, desirable, and well-suited in classroom settings. In this article, we describe the development of these resources.

Methods

Researchers in Norway, the United Kingdom, Uganda, Kenya, and Rwanda collaborated to develop and evaluate learning resources for school children and their parents in 2013 to 2017. This included development of a podcast for parents [14]; development of the CLAIM Evaluation Tool for measuring people’s ability to assess treatment claims [15]; a randomised trial of the effects of using the (IHC) primary school resources [16]; a randomised trial of the effects of listening to the podcast [17]; and a process evaluation [18].

Participants and setting

While most of the piloting, user testing, and feedback took place in Uganda, we wanted to create resources that could also be used in other countries. Therefore, we also carried out piloting and user-testing of Version 2 of the resources in two East African countries (Rwanda and Kenya) and in one high-income country (Norway).

For pilot testing, we recruited schools that were geographically accessible to our team, taught in English, and were willing to make time. We contacted head teachers, who identified science teachers and classes of children who were prepared to pilot the resources. To recruit user-test participants, we used purposeful sampling to include year 5 students (10 to 12-year-olds) and their teachers. Table 1 describes the participants, and (Additional file 1) describes which participants we included in each step of the development work.

Table 1 Participants

Developing the resources

We employed a user-centred approach to designing the IHC primary school resources [19,20,21,22]. User-centred design is characterised by multiple iterative cycles of development (Fig. 1).

Fig. 1
figure 1

User-centred design development in multiple iterative cycles

Our starting point for developing these learning resources was to create a list of 32 Key Concepts that people need to understand and be able to apply to assess claims about treatment effects and make informed health choices [15]. A network of teachers in Uganda assessed the relevance of these concepts for primary school children during the prioritisation process and determined that 24 of these concepts were relevant to primary school children in Uganda [23].

Idea generation and prototyping

We used “creative thinking” in the idea generation and prototyping process. Creative thinking focuses on exploring ideas, generating possibilities and looking for many options [24]. This contrasts with critical assessment, which focuses on analysis, figuring out the answer and eliminating incorrect options. Both types of thinking were necessary for generating appropriate options for the resources we developed.

We needed to bring stakeholders and end users as close as possible into all phases of the work. This was particularly important since none of us belonged to the end user groups we were developing resources for (primary school children and their teachers). We included teachers as close collaborators through brainstorming [25] and prototyping workshops, and sought early feedback from children through workshops and school visits. We conducted multiple workshops in Uganda and Norway with the IHC research team and a network of teachers in Uganda [23]. These workshops resulted in ideas and insights about the context and stakeholders, sketches, and prototypes. We selected and developed ideas that we thought had the most potential to create new prototypes. These prototypes formed the basis for the next phases of pilot testing and user-testing.

Pilot testing and user-testing

We pilot tested early prototypes in workshops with teachers and children and through school visits in Uganda and Norway, using participatory observation to facilitate participants’ engagement. We piloted later, more complete prototypes (Version 1 in Uganda, and Version 2 in Uganda, Rwanda, Kenya and Norway) using non-participatory observations of the classroom lessons to explore how teachers and children used these resources. We used a structured form to record observations (Additional file 2), as well as video and still photography.

We also carried out user-test interviews with individual children and teachers to explore their experience when interacting with our resources [19]. User-testing originated from human computer interaction, where effectiveness and efficiency of a product is measured in relation to personal satisfaction of the individual using the product. We used a qualitative approach, building on Rosenbaum’s adaptation of Peter Moville’s honeycomb framework of user experience [19,20,21,22, 25,26,27] to develop the interview guides. We focused on six facets of the users’ experiences: usefulness, ease of use, understandability, credibility, desirability, and identification (Table 2) [19].

Table 2 Six facets from the honeycomb framework

Analysis and revisions

We used a framework analysis approach to guide data collection and analysis. We entered observations from the pilot testing and feedback from the user-testing into a spreadsheet after each round of testing. Between two and five researchers from the IHC working group independently coded each observation based on the importance of the finding (Table 3) and its implications for changes to the learning resources.

Table 3 Coding of the importance of observations and feedback

The coding was combined in a single spreadsheet, discussed, and a consensus was reached. Based on these findings, we generated a list of potential problems and suggestions for changes. We discussed major problems and brainstormed solutions to those problems with the rest of the IHC working group. After agreeing on the changes needed, we created new prototypes to be piloted and user-tested.

We did not collect or analyse any quantitative data.

A timeline showing the development process, beginning with prioritisation of the Key Concepts is shown in (Fig. 2), and each step is summarised in (Additional file 1).

Fig. 2
figure 2

Development timeline

Patient and public involvement statement

End users such as teachers on the network, policy makers on the advisory panels and primary school children participated in the development process by providing structured feedback of the resources at various iteration stages.

Results

Prioritising key concepts

We started with the list of 32 Key Concepts identified at the beginning of this project [15]. However, although 24 of these concepts were deemed relevant for primary school children, teaching all 24 concepts proved to be too much to learn in a school term. The early prototypes we created had too many concepts per lesson and took too long to teach in a normal school hour (40 min). We also observed that the teachers needed time to repeat material from previous lessons. We therefore reduced the number of concepts addressed in the final version of the resources to 12 (Table 4), as described in (Additional file 3). The other 12 concepts could be taught in a subsequent school term.

Table 4 Key Concepts that are relevant for primary school children

Review, idea generation and exploratory prototypes

This phase lasted two years and was highly exploratory. In addition to the workshops and prototype development described below, we also engaged regularly with the teachers’ network and the Uganda National Advisory board.

Reviewing existing resources

We collected ideas from our own experiences teaching critical appraisal to children [28] and adults (including health professionals, policymakers, journalists, and patients), a systematic review of interactive resources for teaching critical appraisal skills to consumers [29], and searching the TES database and other sources such as google scholar for relevant resources.

We had a series of brainstorming sessions with members of the research team, informed by the resources that we found and workshops that we conducted with teachers and other researchers. In October 2015, we organised an international workshop with others interested in helping people to assess claims about treatments where a variety of resources was discussed. This workshop led to the development of the Critical thinking and Appraisal Resource Library (CARL) [30]. The Critical thinking and Appraisal Resource Library (CARL) is a platform to collect and distribute freely-available learning resources intended to help people think critically about treatment claims.

Idea generation workshop with researchers, teachers and journalists

In this meeting, we generated a broad range of ideas, from holding science fairs to creating interactive videos. Some ideas we generated were: use of drama and storytelling, board and field games, getting children to run a trial over several months, building a collection of familiar examples, translating already existing resources into local languages, holding teacher training workshops.

The main challenges we identified included: the need to teach the teachers; developing resources that would work in schools without digital equipment and where languages other than English were spoken; finding time in the curriculum, and gaining buy-in from stakeholders (including teachers, parents, and policymakers).

We decided to focus the next step on developing interactive classroom games that could be carried out with simple readily available equipment, like blackboards.

Pilot testing games in classrooms

We developed presentation materials and prototypes for two games to be used in classrooms: tossing coins to explain the concept of ‘chance’ and a game involving comparing the effects of two different coloured candies to explain Key Concepts related to fair comparisons. Children worked together in small groups. We piloted the games in classes at three schools – one in Norway and two in Uganda, with numbers of children ranging from 30 to 129. We participated by taking the role of teachers.

The children clearly enjoyed these activities. They were engaged, asked relevant questions and came up with some of the concepts by themselves, like blinding. But the exercise tended to get out of hand when the children were required to work independently and discuss in small groups. This was a problem even in Norway, despite the smaller class size. The children also needed more structured materials and more facilitation than we had anticipated. Their understanding of the concept ‘fair’ was different than what we meant when talking about fair comparisons, which we referred to initially as “fair tests”. One child said:

“For the test to be a fair test, everyone should get a candy”.

Despite being encouraged by the apparent ability of the children to understand many of the concepts, we also experienced first-hand that it could be challenging to explain the concepts correctly, even with semi-structured presentation materials. Teachers who were unfamiliar with the concepts would likely have even more difficulty.

Prototyping and pilot testing in Uganda and Norway

We conducted a prototyping workshop with 24 members of the teachers’ network in Uganda, piloted a game at a school in Uganda and an international school in Norway, and piloted a series of eight games with four 12 year-old girls in Norway (Additional file 4).

We found that although some of the games appeared to be promising, several were still too complicated to carry out in large classrooms. We also still had not solved the problem of how to transfer our presentation role to a teacher who was unfamiliar with the concepts without relying on electronic equipment like PowerPoint or video.

We decided to produce a highly-structured narrative for presenting the Key Concepts, which the teacher and children would read together, as well as a guide for the teacher. We decided to make a narrative in the form of a comic book with game-like activities and individual exercises included. We developed five characters to build the story around: two school children, two professors and a parrot who made unreliable claims about treatments, in an unspecified setting that would look like a rural east African village. Our thinking was that the narrative and use of drawings would engage the children, make the Key Concepts easier to understand, and help them to retain what they learned [31, 32].

Despite many problems, there was enough enthusiasm for the comic format that we felt it had the potential to work in a Ugandan classroom. But it needed to be much simpler, and the explanation of each concept needed more space. Based on our findings and observations, we agreed to make the following changes in the next version:

  • Rewrite and redraw the children’s book with

    • A much simpler story, language, and drawings; and shorter chapters with larger text

    • No complicated comic language

    • Glossary explanations where terms first occur, with definitions translated to Luganda

    • Examples that were less likely to be misleading

    • Simplify all activities so they would not require extra resources, or require being outdoors

  • Revise the teachers’ guide by

    • Making it more like a recipe

    • Integrating the children’s book in the teachers’ guide to facilitate the lesson flow

We decided to produce the final version of the books in colour, but continued sketching prototypes in black-and-white.

The IHC primary school resources

We created three complete versions of the children’s book and teachers’ guide. The first version had 11 chapters (Fig. 3). We carried out pilot tests and user-testing at two schools in Uganda. Based on the users’ experiences (Additional file 5), we made the following changes to the next version of the children’s book:

  • More emphasis on “critical thinking” rather than becoming a “junior researcher”

  • Add a new first chapter that clarifies the purpose of the book, introduces some of the basic vocabulary in more depth (“health” “treatments”, “effects” and “claims”).

  • Make usefulness more apparent by placing the story in the context of real life decision-making (e.g. the children in the book making a poor decision in the beginning and a more informed decision at the end)

  • Adjust chapter content so that lessons could be fitted into 40-min periods

  • Repeat learning goals from the previous chapter and introduce new characters at the beginning of each chapter

  • Continue to simplify vocabulary; add a glossary in the back of the book

  • Use a computer font instead of handwriting

  • Add arrows to the comic cells to indicate reading direction

  • More expressive and differentiated characters

Fig. 3
figure 3

Version 1 prototype of the IHC primary school resources

We agreed on the following changes to the Teachers’ guide:

  • Introduce more structure

  • Add more background information, both about the purpose of the resources and about the key concepts covered in each chapter

  • Decrease the number of lesson goals in each chapter

We created Version 2 of the children’s book and teachers’ guide (Fig. 4), which had 10 chapters divided into two books. We carried out pilot tests and user-testing at schools in Uganda, Rwanda, Kenya, and Norway.

Fig. 4
figure 4

Version 2 prototype of the IHC primary school resources

The most important problem that we identified was insufficient time to teach all the content included in Version 2. Based on the users’ experiences (Additional file 5), we agreed to make the following changes in the next version of the children’s book:

  • Revise the CLAIM game and make it less demanding on the teacher to organise

  • Introduce a glossary that explains all the new terms in the children’s book

  • Reduce the number of exercises at the end of each lesson

  • Further simplify or remove chapters that were difficult for the children to understand, like chapter 8 on “careful summaries” (systematic reviews)

We agreed to make the following changes in the teachers’ guide:

  • Add more examples

  • Revise and restructure the content and add a structured lesson plan

We created Version 3 of the children’s book with 10 chapters, and a teachers’ guide (Fig. 5). We also created a separate exercise book, a classroom poster of the key learning objectives (the 12 Key Concepts), and a set of activity cards for one of the chapters. These open access resources can be viewed or downloaded at http://www.informedhealthchoices.org/primary-school-resources/. The contents of the children’s book and the teachers’ guide are summarised in Table 5.

Fig. 5
figure 5

Version 3 (final) of the IHC primary school resources

Table 5 Contents of the children’s book and the teachers’ guide

Discussion

While focussing on the six facets, (usefulness, ease of use, understandability, credibility, desirability and identification), of Rosenbaum’s adaptation of Peter Moville’s honeycomb frame work of user experience, this work highlights the following lessons for future studies designing educational materials;

Usefulness

Findings from the idea generation and exploratory prototypes phase of the project highlighted the need to clarify the usefulness of the resources for both teachers and children. Teachers’ participating in the Uganda teachers’ network workshop initially assumed that the purpose of the resources was to convey public health messages about the benefits of specific interventions, such as handwashing, healthy eating habits and exercise. Many of the ideas and prototypes generated at that workshop focused on communicating typical public health messages, rather than teaching children to think critically about health claims and choices.

There are several plausible explanations for this. These include that teaching is largely didactic in East Africa, in part due to large student-to-teacher ratios. This makes it difficult to use more interactive teaching strategies required to teach critical thinking skills [33]. Teaching critical thinking skills has not been a priority in primary school curricula or for evaluations of interventions to improve primary school education [12, 34, 35]. Previous public health interventions in schools have also tended to focus on promoting specific behaviours, rather than teaching children to think critically. This contrasts with our findings in Norway. Critical thinking was a priority for older children (in the International Baccalaureate IB programme) at the international school where we piloted the second version of the resources. However, the teachers there found that students entering the IB programme were not sufficiently prepared. They wanted to test our resources specifically to find out if they might help to address this problem that they had already identified.

Expectations of the children in response to early prototypes were different from those of the teachers. They assumed that the purpose of the resources was to help them do better in science and to learn to become scientists or health professionals.

We addressed these misunderstandings about why the resources are useful in several ways. We added introductions to both the children’s book and the teachers’ guide clarifying the purpose of the resources. These went through several iterations and we obtained feedback from teachers and children to ensure that the introductions clarified the purpose of the resources and why they are useful. We ensured that the examples we used would not be misunderstood and that they clearly illustrated how each Key Concept could be used to assess relevant claims and to make informed choices. We modified the structure of the book, and subsequently organised the Key Concepts (from six groups to three groups), to clarify and reinforce the purpose and usefulness of understanding and applying them.

When testing the first and second versions of the resources we found that teachers and most children found the resources useful and correctly understood their purpose by the end of the lessons. In addition to the above changes, we also developed a workshop for teachers to introduce them to the resources and to help ensure that they started out with a clear understanding of the purpose of the resources. The workshop is described in detail in another article [36].

Ease of use

We found that our initial ideas and prototypes were difficult to use, even in well-resourced schools with low student-to-teacher ratios. We also found that many of the Key Concepts were not well understood by the teachers. Frequently they went off script, making unsubstantiated claims themselves rather than helping the children learn how to assess claims. Using a comic book to introduce the Key Concepts solved the problem of ensuring that they were introduced and explained correctly. The illustrations facilitated engagement, understanding and made it easier for the children to read the text. This is consistent with previous research, which has shown that adding pictures to written language can increase attention, comprehension, and recall [32]. However, pictures can also be misunderstood and the feedback we received on the illustrations resulted in many changes - both specific and general. For example, feedback from several children resulted in changes to how Julie, one of the two children who are main characters in the comic book was portrayed. As one child remarked when asked about the drawings in an early version: “Julie looks like a rumour monger.”

We also discovered important changes that were needed to make the comic book usable in Uganda. Many of the children were not familiar with reading comics and were confused about the order in which the frames should be read. They also were not familiar with speech and thought bubbles. To address this problem, we added arrows to the comic, showing the order in which frames should be read and explained speech and thought bubbles in the introduction.

Using a comic book to introduce the Key Concepts functioned well both in East Africa, where it is common for classes to read aloud and in Norway, where role playing was used when reading aloud in class. Based on our observations and interviews, we concluded that there were several ways of reading the book. Rather than recommending one of these, we provided the advice based on what we had observed.

Our observations and feedback from the teachers resulted in several changes to the teachers’ guide to ensure that teachers found it useful. One change was to incorporate the children’s book in the teachers’ guide. This facilitated using the guide, which includes instructions and suggestions for the teachers, while reading the comic together with the children or doing the activities. Others included providing lesson plans, explanations written for the teachers, and extra examples that the teachers could use to illustrate the Key Concepts.

The most important problem that we found with the second version of the resources was insufficient time. Teachers struggled to get though the lessons in 40 min and, therefore, the children were often confused and had not learned some of the Key Concepts. To address this problem, we reduced the number of concepts that were included from 24 to 12 and we doubled the amount of time for each lesson. This required us to step back and acknowledge that we had made a classic mistake of trying to teach everything about a topic at once, thereby overloading both the children and the teachers with too much information. By recognising that the resources are just one cycle in a spiral curriculum [37], we could make this dramatic change. Resources for subsequent cycles can build on what was previously learned from these resources and reinforcing while introducing new concepts.

Other changes that we made to the resources to improve their usability included greatly simplifying the activities to ensure that they could easily be managed by a single teacher with many children and ensuring that the exercises could be done by the children without placing a substantial burden on the teacher.

Understandability

We discussed understanding the purpose of the resources in relation to its perceived usefulness and how that affected the extent to which teachers and children valued the resources. We also found substantial problems with understanding of the content. Many of the children read poorly and English was a second language for most. We found that words that we assumed 10 to 12-year-old children would understand, such as ‘health’, were new words for many children in East Africa. Although using a comic book with illustrations helped to improve understanding, we still needed to further simplify the language that we used explain terms. We addressed this by iteratively testing and rewriting the text, adding a glossary, adding translations of key terms to Luganda and Kiswahili, adding a list of new keywords used in each chapter, and adding explanations and translations of key terms to the text on the page where they were first used (Fig. 6). Together with teachers and children, we also generated a list of terms that were difficult for the children. We avoided using those terms if there was a good alternative or explained them.

Fig. 6
figure 6

Repeating keywords where they first appear in the text

Several changes to the teachers’ guide were made to ensure their understanding, these included adding a background section to each chapter and extra information about the examples that we used (Fig. 7), in addition to the workshop for teachers noted above.

Fig. 7
figure 7

Background section of each chapter for teachers

Credibility

Two problems that we identified were the use of magical elements in the first comic prototype and the inclusion of a talking parrot. We eliminated the former, but elected to keep the parrot for two reasons. First, although teachers were concerned that a talking animal would result in a loss of credibility amongst the children, none of the children perceived this as a problem. Second, the children responded very positively to the parrot, which both brought humour into the story and served as a source of claims. We did, however, review our use of the parrot to ensure that it was used consistently and that it was not included unnecessarily; e.g. repeating something that one of the other characters said.

Desirability

Many of our early ideas, which focused on games, were clearly not something that the teachers wanted. They were difficult to organise and to manage, especially in classes with large student-to-teacher ratios.

We found that the book was highly desirable both in East Africa and in Norway. This was, perhaps, not surprising in East Africa where the schools had few books. However, the children at the international school in Norway also were very positive about the book. They uniformly responded that they would prefer the book to a computer game. It is uncertain to what extent this was because they had been exposed to poorly designed learning games or because the book was well designed. Children in both settings had not previously been exposed to use of a comic book to teach science.

The rationale for using a narrative in the book to explain the Key Concepts is that people often make sense of their lives through stories they hear and share with others [31]. Providing information in a story may resonate with people who might struggle to understand abstract concepts. Furthermore, characters in the narrative can role model new behaviours, enhancing self-efficacy [38]. Evaluations of the effects of narrative interventions support their use. For example, evaluations of the use of narratives in the context of health promotion have found that narrative interventions improve knowledge about health-promoting behaviours and the behaviours themselves [31].

Although we received consistent feedback from the children and teachers that they would prefer resources printed in colour, we also observed that the children clearly enjoyed colouring the prototype line drawings printed without colour. Another problem was that while we had hoped the children would take the books home and share what they were learning with their families, the teachers were worried about the books getting lost and the children not having them in class when they were needed. Our solution to both these problems was to create separate exercise books and textbooks. The final version of the children’s (text) book was in colour, could be kept at school, and could be re-used by other classes. The exercise book (containing key learning goals for each lesson, exercises and a glossary) was printed in black-and white that could be coloured by the children, and could be taken home.

Identification

Initially we received many comments from the children in Uganda about the drawings, particularly about John and Julie, with whom they did not identify. However, with subsequent iterations of the children’s book, the children identified with John and Julie. Similarly, both the teachers and children expressed that the resources felt like they were appropriate for them, increasingly with each iteration.

We were uncertain to what extent children at the international school in Norway would find the characters and the story, which was set in an East African context, relevant to them. To our surprise, we found that some of the children when asked where they thought the setting for the story was did not notice that it was in Africa. Others we spoke to were pleased that the story was set in Africa rather than in North America or Europe, which was the setting for most of the books they used.

Conclusions

Our findings suggest that with the iterative revisions of the IHC primary school resources, users - both children and teachers - experienced the resources as useful, easy to use, understandable, credible, desirable, and well suited for them. We believe there are two closely related reasons why we could achieve this. First, our grant application did not include a specification of what we were going to create. Instead, we described our goals and the methods that we would use to develop resources. This allowed us ample time (two years) to generate and prototype ideas and then to iteratively design, pilot and user-test, analyse, and redesign these resources.

Second, we used a user-centred design approach with a multidisciplinary team and engagement of users throughout the development process. The research team included health service researchers with diverse backgrounds, designers, and a journalist. We collaborated closely with a teachers’ network, a journalists’ network [39], policymakers, and education researchers. We also piloted and user-tested the resources in schools in four countries. This broad range of feedback helped us create resources that increasingly resonated with these diverse communities.

The main limitation to the study was time constraint, in terms of tying the design schedule to the already busy school schedule. This also meant that only schools that were willing to avail time and participated in the development may not have been representative of the larger population. However in a follow up study, we have evaluated the effects of using the resources in a large randomised trial in Uganda (with 120 schools), that showed the intervention led to large improvements in the ability of both the children and their teachers to assess claims about treatments [36]. This trial excluded schools that participated in the development. Therefore, we can be fairly sure that input we gathered from participating schools was also representative for larger groups.

Availability of data and materials

All data will be available on reasonable request. (Extra data can be accessed on https://www.informedhealthchoices.org/learning-resources/).

Abbreviations

CARL:

Critical thinking and Appraisal Resource Library

IB:

International Baccalaureate Programme

IHC:

Informed Health Choices Project

References

  1. Sillience E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health information? Soc Sci Med. 2007;64(9):1853–62.

    Article  Google Scholar 

  2. Irwig L, Irwig J, Trevena L, Sweet M. Smart health choices: making sense of health advice. London: Hammersmith; 2008.

    Google Scholar 

  3. Lloyd AJ. The extent of patients’ understanding of the risk of treatments. Qual Health Care. 2001;10(1):14–8.

    Article  Google Scholar 

  4. Weinstein N. What does it mean to understand a risk? Evaluating risk comprehension. J Natl Cancer Inst Monogr. 1999;25:15–20.

    Article  Google Scholar 

  5. Robinson EJ, Kerr CE, Stevens AJ, Lilford RJ, Braunholtz DA, Edwards SJ, et al. Lay Public's understanding of equipoise and randomisation in randomised controlled trials. Health Technol Assess. 2005;9(8):1–192.

    Article  CAS  Google Scholar 

  6. Austvoll-Dahlgren A, Nsangi A, Semakula D. Interventions and assessment tools addressing key concepts people need to know to appraise claims about treatment effects: a systematic mapping review. Syst Rev. 2016;5:215.

    Article  Google Scholar 

  7. Sandoval WA, Sodian B, Koerber S, Wong J. Developing children’s early compe-tencies to engage with science. Educ Psychol. 2014;49:139–52 Educ Psychol 2014; 49: 139–52. 2014.

    Article  Google Scholar 

  8. Ruddock G, Sainsbury M. Comparison of the core primary curriculum in England to those of other high performing countries. Research Report DCSF-RW048. London: Department for Children, Schools and Families, 2008. Research report DCSF-RW048. London: Department for Children, Schools and Families, 2008; 2008.

    Google Scholar 

  9. Tamusuza A. Leaving School Early, The Quest for Universal Primary Education in Uganda. J Statisque Afr. 2011;11:110–51.

    Google Scholar 

  10. Burke K, Beegle K. Why children Aren’t attending schools, the case of northwestern Tanzania. J Afr Econ. 2004;13(2):333–55.

    Article  Google Scholar 

  11. Deininger K. Does the cost of schooling affect enrollment by the poor? Universal primary education in Uganda. Econ Educ Rev J. 2003;22(3):291–305.

    Article  Google Scholar 

  12. Evans DK, Popova A. What really works to improve learning in developing countries? An analysis of divergent findings in systematic reviews. . Policy Research Working Paper. Washington D. C: World Bank; 2015. p. 7203.

    Google Scholar 

  13. Nordheim L, Gundersen MW, Espehaug B, Guttersrud O, Flottorp S. Effects of school based educational interventions for enhancing adolescents’ abilities in critical appraisal of health claims. a systematic review. PLoS One. 2016;11:0161485.

    Article  Google Scholar 

  14. Semakula D, Nsangi A, Oxman M, Rosenbaum SE, Oxman AD, Austvoll-Dahlgren A, et al. Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the benefits and harms of treatments. BMC Pilot Feasibility Stud. 2019;5:155.

    Article  Google Scholar 

  15. Austvoll-Dahlgren A, Oxman AD, Chalmers I, Nsangi A, Glenton C, Lewin S, et al. Key concepts that people need to understand to assess claims about treatment effects. J Evid Based Med. 2015;8:112–25.

    Article  Google Scholar 

  16. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the informed health choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet. 2017;17:31226.

    Google Scholar 

  17. Semakula D, Nsangi A, Oxman AD, Oxman M, Austvoll-Dahlgren A, Rosenbaum S, et al. Effects of the informed health choices podcast on the ability of parents of primary school children in Uganda to assess claims about treatment effects: a randomised controlled trial. Lancet. 2017;17:31225–4.

    Google Scholar 

  18. Nsangi A, Semakula D, Glenton C, Lewin S, Oxman AD, Oxman M, et al. Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation. BMJ Open. 2019;9(9):e030787.

    Article  Google Scholar 

  19. Rosenbaum S. Improving the user experience of evidence: a design approach to evidence-informed health care. Arkitektur- og designhøgskolen i Oslo: Oslo; 2010.

    Google Scholar 

  20. IDEO. Human Centered Design Toolkit. 1st ed. San Francisco: IDEO; 2011.

    Google Scholar 

  21. Abras C, Maloney-Krichmar D, Preece J. User-Centered Design. In: Bainbridge W, editor. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publi-cations, 2004; 2004.

    Google Scholar 

  22. Sanders E, Stappers P. Co-creation and the new landscapes of design. Co-design. 2008;2008(4):5–18.

    Google Scholar 

  23. Nsangi A, Semakula D, Oxman DA, Sewankambo KN. Teaching children in low income countries to assess claims about treatment effects; A prioritisation of key concepts. J Evid Based Med. 2015;8(4):173–80.

    Article  Google Scholar 

  24. de Bono E. Lateral thinking. A textbook of Creativity. The Penguin Group; 1970.

    Google Scholar 

  25. Hillstrom K, Hillstrom LC. Brainstorming. In: Encyclopedia of small business, vol. 1. Farmington Hills: Gale Cengage; 2002.

    Google Scholar 

  26. Sauro J. A brief history of usability. Denver: MeasuringU; 2013.

    Google Scholar 

  27. Morville P. User Experience Design: honey comb model. Sematic Studios LLC 2004 (Updated June 21, 2004).

    Google Scholar 

  28. Chalmers I. Explaining the unbiased creation of treament comparison groups. Lancet. 2009;374:1671.

    Google Scholar 

  29. Krause L, Schwarts L, Woloshin S, Oxman A. Interactive tools to teach the public to be critical consumers of information about health care: What’s out there? Abstracts of the 19th Cochrane colloquium, Madrid, Spain 157; 2011.

    Google Scholar 

  30. Castle JC, Chalmers I, Atkinson P, et al. Establishing a library of resources to help people understand key concepts in assessing treatment claims- The “Critical thinking and Appraisal Resource Library” (CARL). Plos One. 2017;12:e0178666.

    Article  Google Scholar 

  31. Perrier MJ, Martin Ginis KA. Changing health-promoting behaviours through narrative interventions; a systematic review. J Health Psychol. 2018;23(11):1499–517. https://doi.org/10.1177/1359105316656243.

    Article  Google Scholar 

  32. Houts P, Doak C, Doak L, Loscalzo M. The role of pictures in improving health communication. A review of research on attention, comprehension, recall and adherence. Patient Educ Couns. 2006;62:173–90.

    Article  Google Scholar 

  33. Abrami PC, Bernard RM, Borokhovski E, Waddinton DI, Wade CA, Persson T. Strategies for teaching students to think critically; a meta analysis. Rev Edu Res. 2015;85:275–314.

    Article  Google Scholar 

  34. McEwan PJ. Improving learning in primary schools of developing countries: a meta analysis of randomised experiments. Rev Edu Res. 2015;85:385–94.

    Google Scholar 

  35. Snilstveit B, Stevenson J, Phillips D. al E. interventions for improving learning outcomes and access to education in low and middle-income countries: a systematic review. London: International Initiative for Impact Evaluation; 2015.

    Google Scholar 

  36. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects; A cluster randomised trial. Lancet. 2017;390(10092):374–88 Published online May 21.

    Article  Google Scholar 

  37. Harden RM, Stamper N. What is a spiral curriculum? Med Teach. 1999;21:141–3.

    Article  CAS  Google Scholar 

  38. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84:191–215.

    Article  CAS  Google Scholar 

  39. Semakula D, Nsangi A, Oxman AD, Sewankambo NK. Priority setting for resources to improve the understanding of information about claims of treatment effects in the mass media. J Evid Based Med. 2015;8(2):84–90.

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful for support for this research from the Global Health and Vaccination Research (GLOBVAC) programme of the Research Council of Norway, the English National Institute for Health Research and the James Lind Initiative for supporting IC and the African Academy of Sciences DELTAS Africa Initiative grant # DEL-15-011 to THRiVE-2, with funding from the Wellcome Trust grant # 107742/Z/15/Z for supporting DS and NKS.

Miriam Grønli was responsible for colouring the final version of the children’s resources and Nora Rosenbaum assisted. We are grateful Aisha Hashi, Sara Jaber, Rida Shah, Katie Tveiten, Cynthia Kateme, Mariam Nakato and Ann Babirye who helped test prototypes. Laetiticia Nyirazinyoye and Jenny Moberg helped with piloting and user-testing an earlier version of the learning resources. We want to thank Margaret Nabatanzi and Martin Mutyaba for their help with data collection. We would also like to thank the Informed Health Choices National Advisory Panel and the Teachers’ Network. We are especially grateful to the teachers and children in Uganda, Kenya, Rwanda and Norway who helped with the development, user-testing and piloting of the Informed Health Choices primary school resources and especially the head teachers, the grade five children and teachers, at the following schools who participated in this: Bright Parents’ Junior School, Kitante Primary School, Kisaasi Primary School, Nakivubo Primary School and Wampewo Primary School.

Funding

The Research Council of Norway, Project number 220603/H10, funded the study. The Norwegian Institute of Public Health, recipient of the grant from the Research Council of Norway, is the coordinating centre for the Informed Health Choices project.

Author information

Authors and Affiliations

Authors

Contributions

AN, DS, SR and ADO drafted this article manuscript. All the authors reviewed this article manuscript, provided input, and agreed on this final version. AN and DS were responsible for data collection in Uganda, MK led data collection in Kenya, MM led data collection in Rwanda and MO led data collection in Norway. All the investigators participated in the analyses of findings for each iterative cycle and agreed on consequences of the findings for the next version of the resources. MO drafted all resource texts; ADO and MO were responsible for final text decisions. SR illustrated, designed, and made final design decisions for the children’s book, reminder poster and activity cards; AM designed and made final design decisions for the Teachers’ guide. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Andrew David Oxman.

Ethics declarations

Ethics approval and consent to participate

Teachers who were invited to participate in the pilot testing and user-testing were informed of the purpose of their participation before written permission was obtained. The children were given information about the project to take home for their parents and written permission was obtained for children selected to participate in the user-testing. Consent for the children to participate in the pilot testing was given by the head teachers and teachers. Children and their parents had the same right to refuse participation in piloting the IHC learning resources as they do for the use of any other learning resources used in the schools.

The study was approved by Makerere University Institutional Review Board and the Uganda National Council of Science and Technology as part of the Supporting Informed Healthcare Choices in Low-income Countries Project (Grant no. ES498037) in August 2013. Ethical approval was sought by the IHC project representatives in each of the other countries in compliance with national requirements.

Consent for publication

Written informed consent for publication of data and images was obtained from participants and from parents or guardians of the (children under 16) included in the study.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Overview of the development

Additional file 2.

Structured Observation Form- Chapter one.

Additional file 3.

Prioritisation of Key Concepts.

Additional file 4.

Prototyping.

Additional file 5.

User Experiences of the IHC resources.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nsangi, A., Semakula, D., Rosenbaum, S.E. et al. Development of the informed health choices resources in four countries to teach primary school children to assess claims about treatment effects: a qualitative study employing a user-centred approach. Pilot Feasibility Stud 6, 18 (2020). https://doi.org/10.1186/s40814-020-00565-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40814-020-00565-6

Keywords