Skip to main content
Download PDF
Download ePub

Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process

Implementation Science volume 11, Article number: 98 (2015) Cite this article

Abstract

Background

In 2012, the World Health Organization (WHO) published recommendations on the use of optimization or “task-shifting” strategies for key, effective maternal and newborn interventions (the OptimizeMNH guidance). When making recommendations about complex health system interventions such as task-shifting, information about the feasibility and acceptability of interventions can be as important as information about their effectiveness. However, these issues are usually not addressed with the same rigour. This paper describes our use of several innovative strategies to broaden the range of evidence used to develop the OptimizeMNH guidance. In this guidance, we systematically included evidence regarding the acceptability and feasibility of relevant task-shifting interventions, primarily using qualitative evidence syntheses and multi-country case study syntheses; we used an approach to assess confidence in findings from qualitative evidence syntheses (the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach); we used a structured evidence-to-decision framework for health systems (the DECIDE framework) to help the guidance panel members move from the different types of evidence to recommendations.

Results

The systematic inclusion of a broader range of evidence, and the use of new guideline development tools, had a number of impacts. Firstly, this broader range of evidence provided relevant information about the feasibility and acceptability of interventions considered in the guidance as well as information about key implementation considerations. However, inclusion of this evidence required more time, resources and skills. Secondly, the GRADE-CERQual approach provided a method for indicating to panel members how much confidence they should place in the findings from the qualitative evidence syntheses and so helped panel members to use this qualitative evidence appropriately. Thirdly, the DECIDE framework gave us a structured format in which we could present a large and complex body of evidence to panel members and end users. The framework also prompted the panel to justify their recommendations, giving end users a record of how these decisions were made.

Conclusions

By expanding the range of evidence assessed in a guideline process, we increase the amount of time and resources required. Nevertheless, the WHO has assessed the outputs of this process to be valuable and is currently repeating the approach used in OptimizeMNH in other guidance processes.

Peer Review reports

Background

In 2012, the World Health Organization (WHO) published evidence-based recommendations regarding the use of optimization or “task-shifting” strategies for key, effective maternal and newborn interventions (OptimizeMNH) [1]. Typically, task-shifting involves training lower level health worker cadres to deliver tasks that are normally delivered by higher level health worker cadres. The main aim of this strategy is to increase access to care for people living in areas with a shortage of health workers.

While task-shifting strategies have the potential to increase access to care in settings with critical shortages of specialised health workers, there is also concern that the delivery of health care by health workers with less training could cause harm to people already in vulnerable situations. The effectiveness and safety of such interventions therefore needs to be rigorously evaluated. However, task-shifting strategies that have been assessed as effective and safe in the context of experimental studies may be challenging to implement at scale within a health system. The shifting of tasks from one group of health workers to another can involve social, behavioural and organisational change and, like other complex interventions, its success may depend not only on characteristics of the intervention itself but also on characteristics of the systems in which the intervention is implemented or on interactions between properties of the intervention and systems [2–4]. The success of task-shifting strategies is also likely to be influenced by factors such as the training, supervision and support provided to health workers as well as access to supplies and referral systems. The culture and organisation of healthcare delivery in a particular setting, as well as views and expectations among stakeholders such as service users, the providers involved in task-shifting and managers, may also play important roles in the success of these types of strategies [5–7]. Furthermore, the shifting of tasks can have serious implications for health worker motivation, for instance because of changes in working conditions and salaries, and for health service users’ experience of care [8, 9].

As these issues can represent important challenges to successful implementation, information about an intervention’s feasibility and acceptability can therefore be as important as information regarding intervention effectiveness and safety when recommendations are made. Initiatives that aim to improve the quality of guidelines, such as the AGREE II instrument, also emphasise the importance of seeking stakeholders’ views and preferences, and of discussing potential organisational barriers to the application of recommendations [10]. But despite their potential importance, questions about acceptability and feasibility are usually not addressed with the same rigour as questions about intervention effectiveness. Instead, it appears that these issues are often left to the opinions and experiences of guideline panel members. While this input is useful as part of a dialogue [11], it is limited by the composition of the group. For global guidelines such as OptimizeMNH, the input of panel members from specific geographic or clinical settings and with experience of particular population groups may not give sufficient insight into the values and experiences of service users and other stakeholders more broadly. In addition, the perspectives of some panel members may take precedence over other members because of perceived differences in social status within the guideline panel [12, 13].

As the technical team responsible for developing the OptimizeMNH guidance (Table 1), we saw a need to address these issues in the evidence-to-decision process. In this paper, we describe our use of innovative strategies to expand the evidence used as a basis for WHO recommendations and discuss our experiences with these strategies.

Table 1 Defining guidance
Full size table

Methods

In developing the OptimizeMNH guidance, we followed the standard procedures set out in the WHO’s Handbook for Guideline Development [14]. These procedures (which include the formulation of questions and outcomes; the retrieval, assessment and synthesis of evidence; the development and formulation of recommendations; and finally, the dissemination and implementation of these recommendations) are intended to ensure that WHO recommendations address key needs and are evidence-based [15]. However, we also elaborated on these standard procedures by adopting innovative approaches that could help us address our goal of producing evidence-informed recommendations that are informed by a broader set of evidence than data on effectiveness alone. These approaches were used at different stages of guidance development: when gathering the evidence to support the guidance, when assessing and synthesising this evidence, and when developing the recommendations (Table 2). The approaches included the following:

Table 2 Innovative strategies and resources available to guidance developers
Full size table

  • Broadening the range of evidence used in the guidance by including systematic reviews of qualitative research, multi-country case study syntheses and results from a primary qualitative study

  • Using the Grading of Recommendations, Assessment, Development and Evaluation-Confidence in Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach to assess our confidence in the evidence from systematic reviews of qualitative research

  • Using a structured health systems framework (the DECIDE framework) to help the guidance panel move from this wider range of evidence to recommendations

Results

Gathering the evidence—broadening the range of evidence included in the guidance

The WHO’s Handbook for Guideline Development describes not only several types of questions that can legitimately be addressed in a WHO guideline, including the effectiveness or efficacy of an intervention, but also the negative consequences of the intervention, its social acceptability and its cost-effectiveness. The Handbook describes how qualitative evidence synthesis can be used to explore contextual barriers as well as values and preferences [14]. The main focus of the Handbook is, however, on the use of evidence about effectiveness and, in practice, most WHO guidelines have focused almost exclusively on this question.

For OptimizeMNH, we systematically assessed the effectiveness and safety of the relevant task-shifting interventions through systematic reviews and the use of GRADE. In addition, we used an innovative approach in this stage of guidance development by systematically including other types of evidence. Specifically, we compiled evidence on the acceptability of the interventions among health workers, recipients of care, policy makers and other stakeholders and on the feasibility of these interventions, taking into account the organisational changes to health systems that each task-shifting intervention would entail.

While systematic reviews of randomised trials are seen as the most robust source of evidence when assessing intervention effectiveness, this approach is not well-suited for exploring issues surrounding intervention acceptability and feasibility, particularly for more complex health and social interventions [2, 16]. To answer these questions, we included systematic reviews of qualitative studies (sometimes called qualitative evidence syntheses) as our primary source of evidence [17]. Together with colleagues, we co-authored four qualitative evidence syntheses focusing on factors affecting the implementation of task-shifting among lay health workers, midwives, doctors and nurses [18–21].

The qualitative evidence syntheses provided us with information that was clearly relevant to the questions we had asked. However, most of the studies were of relatively small-scale or pilot programmes, often implemented in the context of research, and focused on factors unfolding at the level of programme delivery in communities and primary care facilities. We decided therefore to complement this evidence with two multi-country case study syntheses focusing on factors affecting the implementation of large-scale programmes. When selecting these programmes, we aimed for broad geographical representation and looked for countries in Africa, Asia and South America that had implemented large-scale programmes. To make the synthesis manageable, we limited ourselves to seven programmes and only selected programmes where a reasonable amount of English-language documentation was available. The multi-country case study syntheses aimed to identify “upstream”, system-level factors associated with programme policies, governance, financing, planning, management and organisation [1, 22]. These case studies synthesised evidence from a variety of sources, including peer-reviewed qualitative and quantitative studies, programme reports, web sites and information from personal communication with individuals familiar with the selected task-shifting programmes.

Another limitation of the evidence we identified through the qualitative evidence syntheses was that it focused primarily on the views and experiences of service users and healthcare providers but paid less attention to the viewpoints of programme managers, policy makers and other stakeholders. We attempted to address this imbalance by carrying out our own primary study. We undertook qualitative analyses of the views and experiences of those contributing to the “Health Information For All By 2015” electronic discussion list, which covers a diverse membership of programme managers, healthcare providers, policy makers, academics and others from 170 countries [23]. These analyses evaluated opinions about how the roles of healthcare providers could be optimised to improve maternal and newborn health in LMICs and the implications of such role optimization [1, 24].

When presenting the evidence from the qualitative evidence syntheses, the country case studies and the primary qualitative studies, we attempted to extract findings that were relevant to specific recommendations being considered by the guidance panel and that concerned specific tasks and groups of health workers. These findings were used to (a) support the guidance panel when reaching recommendations and (b) develop information for end users of the guidance regarding implementation considerations for each recommendation. Tables 3 and 4 provide examples of how these findings informed the development of recommendations in the OptimizeMNH guidance.

Table 3 Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (1): provision of continuous support during labour by lay health workers, in the presence of a skilled birth attendant
Full size table
Table 4 Example of how the qualitative evidence informed the final recommendations in the OptimizeMNH guidance (2): provision of vasectomy by trained midwives
Full size table

We also wanted to offer those using the OptimizeMNH guidance more general information about the acceptability and feasibility of task-shifting strategies. We therefore carried out a “cross-cutting” analysis of findings from all of the sources described above. This cross-cutting analysis provided evidence regarding task-shifting in general where no direct evidence was available for specific task-shifting interventions. The cross-cutting analysis also contributed to a chapter in the WHO OptimizeMNH guidance on overarching implementation considerations.

Assessing and synthesising the evidence—introducing an approach to assess our confidence in findings from qualitative evidence syntheses

The WHO Guideline Handbook notes that guideline technical teams need to provide an assessment of the quality of the evidence included in a WHO guideline [14]. For evidence of effectiveness, the WHO uses the GRADE approach to assess quality [25]. GRADE is not, however, designed to be applied to qualitative research, and we were not aware of other systems for assessing quality or confidence in findings from qualitative evidence syntheses or for indicating these assessments to end users. Having made the decision to include qualitative evidence syntheses, we therefore needed an approach in transparently assessing and describing how much confidence to place in findings from these types of syntheses.

Work to develop this approach was carried out in collaboration with other qualitative researchers [26], informed by the principles of qualitative research. We were also influenced by the processes used by the GRADE Working Group, which has given considerable thought to how to assess confidence in evidence from systematic reviews. This work resulted in an approach which we named GRADE-CERQual [26]. Since its initial use in the OptimizeMNH guidance, CERQual is being developed further as part of the GRADE Working Group and is now being used in several other WHO guidance development processes [27].

Developing the recommendations: using a structured health systems framework

When assessing the different factors that influence recommendations, including effectiveness, acceptability, feasibility, and cost, the Guideline Handbook [14] suggests that the guideline panel makes use of evidence-to-recommendation tables. These tables can be used to lay out what we know about these different factors and can also be used to record the guideline panel’s judgements about these factors and how they contributed to the development of the recommendation [14]. For OptimizeMNH, we piloted an early paper version of a decision table developed by the DECIDE project [28] and the Grade Working Group [29]: the DECIDE Evidence to Decision framework [30, 31].

The DECIDE framework aims to help guidance panel members move from evidence to health system recommendations by informing judgements about the balance of consequences of each option. The framework is based on a review of relevant literature, brainstorming, feedback from stakeholders [32], and the application of the framework to examples.

For each guidance question, the framework presented, in a structured format, a summary of the evidence regarding the following considerations:

  • The benefits and harms of the intervention/s (sometimes referred to as “effectiveness and safety”)

  • Anticipated resource use

  • Acceptability of the intervention/s

  • Feasibility of the intervention/s

For each of these elements, the framework also included an assessment of the certainty of the evidence, using GRADE (for evidence of effectiveness) and CERQual (for evidence on acceptability and feasibility from qualitative evidence syntheses). The following were also included for each guidance question:

  • A judgement regarding the balance of desirable and undesirable consequences for the intervention/s

  • A recommendation and a justification for this

  • Implementation considerations for the intervention/s

  • Relevant monitoring and evaluation/research priorities in relation to the intervention/s

In addition to the evidence collated in the DECIDE frameworks, full evidence profiles for the reviews of the effectiveness of interventions, as well as summaries of findings for the qualitative evidence syntheses on the acceptability, feasibility and implementation of these interventions, were made available to the guidance panel. These summaries were, in turn, linked to full systematic reviews. The DECIDE Team also prepared a video to help guidance panel members understand the purpose of the framework and how to use it. This was made available prior to the guidance panel meeting and also shown at the start of the meeting.

Discussion

Broadening the range of evidence included in guidance: lessons learnt

The first innovative approach that we used in the development of the OptimizeMNH guidance was the systematic inclusion of additional types of evidence to complement the evidence of effectiveness (Table 5). This approach had a number of implications. On a practical level, broadening the scope of evidence to be considered in relation to each guidance question required more time and resources. It also required a wider set of synthesis, assessment and interpretation skills within the technical team preparing the documents for the guidance panel. In our case, these skills were represented within the technical team. Other technical teams using a similar guidance development approach may not have the skills to undertake qualitative evidence syntheses or country case study syntheses but will at least need skills in commissioning and critically appraising such evidence. Creating technical teams with these skills may be challenging as there are few groups with extensive experience in implementing them. However, the number of researchers with the appropriate skills is increasing rapidly, and guidance is now well-developed [33–35].

Table 5 Overview of the different types of evidence used in the OptimizeMNH guidance development process
Full size table

We had originally planned to gather evidence about stakeholder acceptability and feasibility through qualitative evidence syntheses only. As described above, these syntheses, which only included published qualitative studies, primarily offered evidence surrounding factors and stakeholders at the community or primary healthcare facility level. This led us to carry out the multi-country case studies, where we included a broader set of information sources, and also led us to carry out our own primary study among stakeholders poorly represented in the qualitative evidence syntheses. In retrospect, we think that the findings of this primary study should have been included in the qualitative evidence syntheses rather than being treated as a stand-alone product.

Although this innovation expanded the preparatory work needed for the guidance, we would argue that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses offered relevant information for the guidance panel on the feasibility and acceptability of the interventions. When such evidence is not systematically assessed and included, acceptability and feasibility issues may either be overlooked altogether or incorporated into guideline decisions based only on anecdotal evidence. Our experience from the development of the OptimizeMNH guidance suggests that the inclusion of evidence from qualitative evidence syntheses and country case study syntheses reduced the use of anecdotal evidence by guidance panel members when assessing the balance of consequences for each guidance question. In addition, our use of such evidence appeared to address guideline panel members’ perceptions that the WHO often over-emphasises evidence from randomised trials at the expense of evidence from programme experience. One additional benefit of synthesising relevant qualitative evidence was that we were able to use this to develop implementation considerations for task-shifting in maternal and newborn health [1].

Introducing an approach to assess confidence in findings from qualitative evidence syntheses: lessons learnt

Our second innovation was the use of the GRADE-CERQual approach in order to systematically and transparently assess our confidence in findings from the qualitative evidence syntheses conducted to inform the guidance.

Our use of CERQual was received well by the guidance panel, in part because it addressed the need to systematically incorporate a wider range of evidence into the guidance process. The approach also had a number of advantages. When making recommendations, all guidance panels need to take into account how confident they can be in the underlying evidence. Had we not provided the OptimizeMNH panels with the CERQual assessments of our confidence in the findings from the qualitative evidence syntheses, they would have had to make their own judgments and may not have done this in a systematic way. The use of CERQual also allowed us to identify and highlight gaps in the evidence. For instance, where we assessed the confidence in the evidence for a qualitative finding to be low or very low, this was an indication that further research was needed in this area. These research gaps were also reflected in the information presented to the guidance panel and carried through to the recommendations.

We were initially concerned that using the CERQual approach would confuse guidance panel members as they would not have encountered this approach before. In practice, this did not appear to be a problem, probably because members were already familiar with GRADE for assessing findings regarding the effectiveness of interventions, which uses the same principles, and because the panel was briefed on the approach at the start of the meetings.

Some of the findings contributing to OptimizeMNH were based on a very wide range of evidence types. In particular, the country case studies included both qualitative and quantitative studies as well as programme reports, which often did not provide detailed descriptions of methods used. Using CERQual to assess our confidence in the findings from the country case study syntheses was not feasible as the CERQual approach is not designed, at present, to accommodate this range of evidence [26].

Using a structured health systems framework: lessons learnt

Our final innovation was the use of DECIDE evidence-to-decision frameworks [36] to present evidence to the guidance panel. There were several advantages in using this structured framework. Firstly, the framework helped panel members to think through each of the considerations that might be important in making a recommendation and ensured that key considerations were taken into account. The concise format also appeared to help focus and structure the discussions and may have improved the use of the limited time that the guidance panel had to weigh the balance of consequences and make judgements about complex questions regarding task-shifting. Because the framework prompted the guidance panel to justify all of the recommendations made, we believe that the transparency of the decision-making process was improved. Furthermore, end users of the guidance have access to a clear record of recommendation decisions and how these were made. The framework also contributes to closing the knowledge-to-action cycle by capturing implementation considerations and highlighting evidence gaps where more research is needed [37, 38].

The use of the evidence-to-decision frameworks also led to a number of challenges. First of all, preparation of the frameworks took considerable time in advance of the panel meeting, although this may have saved time later on when preparing the final guidance document. Secondly, because the DECIDE framework is a new tool, it required some explanation in advance of the panel meetings. However, panel members appeared to grasp the function and content quickly, and the informal feedback on the framework was positive.

Because the OptimizeMNH guidance included a very large number of recommendations (128 in total), the amount of time available in the panel meetings to discuss each evidence-to-decision framework was limited. The inclusion of additional types of evidence also added to the amount of information that panel members were expected to read. Nonetheless, our experience suggests that the structured format of the DECIDE framework allowed the technical team to present a large and complex body of evidence to panel members in a fairly straightforward and easy-to-assimilate way. Further work is now needed to explore guidance panels’ views of such evidence-to-decision frameworks and to examine how the presentation of evidence in these formats impacts on the deliberations and decisions of guidance panels and on the transparency to users of these decisions.

Conclusion

In this paper, we have described our use of a number of innovative strategies to expand the range of evidence used to develop WHO global recommendations and discussed our experiences with these strategies. Expanding the range of evidence contributing to a guidance process increases the amount of time and resources needed and the range of skills required within the technical team developing the guidance. However, our experience with the OptimizeMNH guidance suggests that such efforts may be valuable, particularly for recommendations on more complex health and social interventions. This view is supported by efforts within WHO to replicate the OptimizeMNH approach in other guidance processes [27]. When considering whether to adopt a similar approach, producers of guidance should assess the extent to which acceptability and feasibility issues are likely to be important enough to influence their recommendations. So far, the WHO is using this approach in recommendations in guidance on more complex health systems and behavioural interventions, but acceptability and feasibility issues may be equally relevant for clinical interventions. In fact, we would suggest that there are few situations where these issues are not likely to be relevant, and even in situations where guidance developers feel confident that they already have a sufficient overview of these issues and additional evidence is not required, a framework approach where this is made transparent can still be useful. We have developed a list of questions (Table 6) to help groups involved in developing guidance decide whether additional evidence syntheses are needed. Further research is needed to refine these questions and to develop an understanding of the typical time and resources required to expand the range of evidence sources used to develop guidance. Table 2 also provides links to resources available to guidance developers who are considering expanding the range of syntheses that will be used to inform the guidance.

Table 6 When to consider expanding the range of evidence syntheses undertaken to inform the development of a guideline or guidance
Full size table

As more qualitative evidence syntheses are produced, methods for synthesising this type of evidence are becoming more sophisticated and the number of people able to produce these types of syntheses is growing. These developments will make the process of including qualitative evidence in evidence-to-decision processes easier in the future. However, a number of challenges remain, as discussed above. Elsewhere, we have described how evidence from qualitative research can be used in developing guidelines [39], including to help shape a guideline’s key questions by informing the populations, interventions, comparators and outcomes on which each key question should focus and to understand how different stakeholders value different outcomes. We did not use these additional strategies in OptimizeMNH, but they have been part of the development for a forthcoming WHO guideline on antenatal care [40].

Our experience from OptimizeMNH suggests that relevant programmatic experience regarding “upstream” systems-level factors is often unexplored or undocumented using traditional research approaches. Researchers should explore these issues to a greater extent [22]. Alongside this, we need to develop better methods for identifying and synthesising information from programme reports and other grey literature as well as methods for assessing our confidence in syntheses based on this type of data [41, 42].

The OptimizeMNH experience suggests that the DECIDE evidence-to-decision frameworks are a useful way of guiding panels through a wide range of evidence and towards the development of transparent recommendations. We have learnt a great deal about how to populate these frameworks with evidence of the effectiveness, feasibility and acceptability of health interventions. However, more research is needed, in particular on how these kinds of evidence are used by guidance panels in their decision-making and how different kinds of evidence influence the final recommendations made. In addition, we need further worked examples of approaches for including evidence regarding resource use and cost-effectiveness. This is one area that those involved in the DECIDE work are exploring [36].

Finally, we need to explore how this type of health system guidance can best be disseminated to and utilised by end users, including national and regional policy makers and programme planners. For most health systems questions, evidence regarding effectiveness, acceptability, feasibility, and resource use is all likely to be context-specific to some degree. End users therefore need help in adapting the recommendations to their own contexts. The OptimizeMNH guidance includes a workbook that specifically aims to help end users contextualise the recommendations [43]. However, more work is needed on tools to contextualise global and national research and guidance for implementation in specific settings [44–47] and on ways of appraising such guidance [48].

References

  1. WHO. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting (OptimizeMNH). Geneva: World Health Organization; 2012.

    Google Scholar 

  2. Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66(11):1262–70. doi:10.1016/j.jclinepi.2013.07.003.

    Article  PubMed  Google Scholar 

  3. Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5(9):e008592. doi:10.1136/bmjopen-2015-008592.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:63. doi:10.1186/1741-7015-8-63.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Dambisya YM, Matinhure S. Policy and programmatic implications of task shifting in Uganda: a case study. BMC Health Serv Res. 2012;12:61. doi:10.1186/1472-6963-12-61.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Georgeu D, Colvin CJ, Lewin S, Fairall L, Bachmann MO, Uebel K, et al. Implementing nurse-initiated and managed antiretroviral treatment (NIMART) in South Africa: a qualitative process evaluation of the STRETCH trial. Implement Sci. 2012;7:66. doi:10.1186/1748-5908-7-66.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Ledikwe JH, Kejelepula M, Maupo K, Sebetso S, Thekiso M, Smith M, et al. Evaluation of a well-established task-shifting initiative: the lay counselor cadre in Botswana. PLoS One. 2013;8(4):e61601. doi:10.1371/journal.pone.0061601.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Cataldo F, Kielmann K, Kielmann T, Mburu G, Musheke M. ‘Deep down in their heart, they wish they could be given some incentives’: a qualitative study on the changing roles and relations of care among home-based caregivers in Zambia. BMC Health Serv Res. 2015;15:36. doi:10.1186/s12913-015-0685-7.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Glenton C, Scheel IB, Pradhan S, Lewin S, Hodgins S, Shrestha V. The female community health volunteer programme in Nepal: decision makers’ perceptions of volunteerism, payment and other incentives. Soc Sci Med (1982). 2010;70(12):1920–7. doi:10.1016/j.socscimed.2010.02.034.

    Article  Google Scholar 

  10. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–42. doi:10.1503/cmaj.090449.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S14. doi:10.1186/1478-4505-7-s1-s14.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Atkins L, Smith JA, Kelly MP, Michie S. The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside. Implement Sci. 2013;8:101. doi:10.1186/1748-5908-8-101.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Pagliari C, Grimshaw J. Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002;8(2):145–53.

    Article  PubMed  Google Scholar 

  14. WHO. World Health Organization handbook for guideline development. 2nd ed. Geneva: World Health Organization; 2014.

    Google Scholar 

  15. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369(9576):1883–9. doi:10.1016/s0140-6736(07)60675-8.

    Article  PubMed  Google Scholar 

  16. Petticrew M, Rehfuess E, Noyes J, Higgins JP, Mayhew A, Pantoja T, et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute. J Clin Epidemiol. 2013;66(11):1230–43. doi:10.1016/j.jclinepi.2013.06.005.

    Article  PubMed  Google Scholar 

  17. Colvin CJ. Anthropologies in and of evidence making in global health research and policy. Med Anthropol. 2015;34(2):99–105. doi:10.1080/01459740.2014.963196.

    Article  PubMed  Google Scholar 

  18. Glenton C, Khanna R, Morgan C, Nilsen ES. The effects, safety and acceptability of compact, pre-filled, autodisable injection devices when delivered by lay health workers. Tropical Med Int Health. 2013;18(8):1002–16. doi:10.1111/tmi.12126.

    Article  Google Scholar 

  19. Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, et al. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev (Online). 2013;10:Cd010414. doi:10.1002/14651858.CD010414.pub2.

    Google Scholar 

  20. Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, et al. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013. doi:10.1016/j.midw.2013.05.001.

    PubMed  Google Scholar 

  21. Rashidian A, Shakibazadeh E, Karimi- Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (Protocol). Cochrane Database Syst Rev. 2013(2). doi: 10.1002/14651858.CD010412.

  22. Gopinathan U, Lewin S, Glenton C. Implementing large-scale programmes to optimise the health workforce in low- and middle-income settings: a multicountry case study synthesis. Tropical Med Int Health. 2014;19(12):1437–56. doi:10.1111/tmi.12381.

    Article  Google Scholar 

  23. (HIFA2015) HIFAB. Health information for all by 2015 electronic discussion list. Accessed 2nd February 2016. http://www.hifa2015.org/.

  24. Owolabi OO, Glenton C, Lewin S, Pakenham-Walsh N. Stakeholder views on the incorporation of traditional birth attendants into the formal health systems of low-and middle-income countries: a qualitative analysis of the HIFA2015 and CHILD2015 email discussion forums. BMC Pregnancy Childbirth. 2014;14:118. doi:10.1186/1471-2393-14-118.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. doi:10.1016/j.jclinepi.2010.04.026.

    Article  PubMed  Google Scholar 

  26. Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, et al. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895. doi:10.1371/journal.pmed.1001895.

    Article  PubMed  PubMed Central  Google Scholar 

  27. WHO. Health worker roles in providing safe abortion care and post abortion contraception. Geneva: World Health Organization; 2015.

    Google Scholar 

  28. DECIDE. Developing and evaluating communication strategies to support informed decisions and practice based on evidence. Accessed 2nd February 2016. www.decide.collaboration.eu.

  29. GRADE. The GRADE working group. Accessed 2nd February 2016. www.gradeworkinggroup.org.

  30. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, GRADE Working Group. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction. BMJ (Clin Res Ed). 2016;353:i2016. doi:10.1136/bmj.i2016.

  31. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl E, Davoli M, Grade Working Group. GRADE Evidence to Decision (EtD) frameworks: 2. clinical practice guidelines. BMJ (Clin Res Ed). 2016;353:i2089. doi:10.1136/bmj.i2089.

  32. Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gulmezoglu AM, et al. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst. 2013;11:19. doi:10.1186/1478-4505-11-19.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Gough D, Oliver S, Thomas J. An introduction to systematic reviews. Thousand Oaks: Sage Publications; 2012.

    Google Scholar 

  34. Noyes J, Popay J, Pearson A, Hannes K, Booth A, on behalf of the Cochrane Qualitative Research Methods Group. Chapter 20: qualitative research and Cochrane reviews. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.www.cochrane.org.

  35. Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: University Press; 2012.

  36. Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. doi:10.1186/1748-5908-8-6.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24. doi:10.1002/chp.47.

    Article  PubMed  Google Scholar 

  38. Kitson A, Straus SE. The knowledge-to-action cycle: identifying the gaps. CMAJ. 2010;182(2):E73–7. doi:10.1503/cmaj.081231.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Glenton C, Lewin S, Norris S. Using evidence from qualitative research to develop WHO guidelines. In: Norris S, editor. Handbook for Guideline Development. 2nd ed. Geneva: WHO: World Health Organization; 2016.

    Google Scholar 

  40. Downe S, Finlayson K, Tuncalp Ӧ, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39. doi:10.1111/1471-0528.13819.

    Article  CAS  PubMed  Google Scholar 

  41. Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187. doi:10.1371/journal.pmed.1001187.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Yin R. Case study research: design and methods. Los Angeles: Sage Publications; 2009.

    Google Scholar 

  43. WHO. Annex 8: contextualizing the guidelines—workbook. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting (OptimizeMNH). Geneva: World Health Organization; 2012.

    Google Scholar 

  44. Bergstrom A, Skeen S, Duc DM, Blandon EZ, Estabrooks C, Gustavsson P, et al. Health system context and implementation of evidence-based practices-development and validation of the Context Assessment for Community Health (COACH) tool for low- and middle-income settings. Implement Sci. 2015;10:120. doi:10.1186/s13012-015-0305-2.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Maar M, Yeates K, Barron M, Hua D, Liu P, Moy Lum-Kwong M, et al. I-RREACH: an engagement and assessment tool for improving implementation readiness of researchers, organizations and communities in complex interventions. Implement Sci. 2015;10:64. doi:10.1186/s13012-015-0257-6.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Burford B, Lewin S, Welch V, Rehfuess E, Waters E. Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making. J Clin Epidemiol. 2013;66(11):1251–61. doi:10.1016/j.jclinepi.2013.06.017.

    Article  PubMed  Google Scholar 

  47. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. doi:10.1186/1748-5908-8-35.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Ako-Arrey DE, Brouwers MC, Lavis JN, Giacomini MK. Health systems guidance appraisal—a critical interpretive synthesis. Implement Sci. 2016;11(1):9. doi:10.1186/s13012-016-0373-y.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Rottingen JA, Droschel D, et al. Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9(3):e1001185. doi:10.1371/journal.pmed.1001185.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Gilson L. Qualitative research synthesis for health policy analysis: what does it entail and what does it offer? Health Policy Plan. 2014;29 Suppl 3:iii1–5. doi:10.1093/heapol/czu121.

    Article  PubMed  Google Scholar 

  51. Hodnett ED, Gates S, Hofmeyr GJ, Sakala C, Weston J. Continuous support for women during childbirth. Cochrane Database Syst Rev (Online). 2011(2):Cd003766. doi:10.1002/14651858.CD003766.pub3.

  52. Polus S, Lewin S, Glenton C, Lerberg PM, Rehfuess E, Gulmezoglu AM. Optimizing the delivery of contraceptives in low- and middle-income countries through task shifting: a systematic review of effectiveness and safety. Reprod Health. 2015;12:27. doi:10.1186/s12978-015-0002-2.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Thanks to Heather Munthe-Kaas, Benedicte Carlsen, Chris Colvin, Jane Noyes, Andrew Booth, Ruth Garside, Arash Rashidian, Meghan Bohren, Özge Tuncalp and Megan Wainwright for their contribution to the development of the GRADE-CERQual approach and to Jenny Moberg, Sarah Rosenbaum and Andy Oxman for their support in the use of the DECIDE frameworks. Thanks also to all those who contributed to the reviews and other materials produced for OptimizeMNH and those who peer reviewed those documents and to Susan Norris from WHO’s Guideline Review Committee Secretariat for her support and encouragement throughout this work.

CG and SL were funded by the Norwegian Agency for Development Corporation (Norad) and by the Norwegian Knowledge Centre for the Health Services (NOKC). SL is also supported by funding from the South African Medical Research Council. None of these funders had any involvement in the design, collection, analysis or interpretation of data, in manuscript preparation or submission. This work was also supported by funding from the UNDP/UNFPA/ UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, Department of Reproductive Health and Research, WHO, Geneva, Switzerland.

Authors’ contributions

All authors conceived of the article, participated in its design and helped to draft the manuscript. All authors read and approved the final manuscript.

Authors’ information

CG, SL and MG are all active contributors to the work of the Cochrane Collaboration.

Competing interests

The authors have all participated in the development of the DECIDE evidence-to-decision frameworks and the GRADE-CERQual approach.

Ethics approval and consent to participate

Not applicable. This study did not involve any humans or animals.

Author information

Authors and Affiliations

  1. Global Health Unit, Knowledge Centre for the Health Services, Norwegian Institute of Public Health, Oslo, Norway

    Claire Glenton & Simon Lewin

  2. Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa

    Simon Lewin

  3. Department of Reproductive Health and Research, including the UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), World Health Organization, Geneva, Switzerland

    Ahmet Metin Gülmezoglu

Authors
  1. Claire Glenton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Lewin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmet Metin Gülmezoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claire Glenton.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Glenton, C., Lewin, S. & Gülmezoglu, A.M. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process. Implementation Sci 11, 98 (2015). https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-016-0470-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-016-0470-y

Keywords

Implementation Science

ISSN: 1748-5908