Stakeholder involvement in systematic reviews: a scoping review

Abstract

Background

There is increasing recognition that it is good practice to involve stakeholders (meaning patients, the public, health professionals and others) in systematic reviews, but limited evidence about how best to do this. We aimed to document the evidence-base relating to stakeholder involvement in systematic reviews and to use this evidence to describe how stakeholders have been involved in systematic reviews.

Methods

We carried out a scoping review, following a published protocol. We searched multiple electronic databases (2010–2016), using a stepwise searching approach, supplemented with hand searching. Two authors independently screened and discussed the first 500 abstracts and, after clarifying selection criteria, screened a further 500. Agreement on screening decisions was 97%, so screening was done by one reviewer only. Pre-planned data extraction was completed, and the comprehensiveness of the description of methods of involvement judged. Additional data extraction was completed for papers judged to have most comprehensive descriptions. Three stakeholder representatives were co-authors for this systematic review.

Results

We included 291 papers in which stakeholders were involved in a systematic review. Thirty percent involved patients and/or carers. Thirty-two percent were from the USA, 26% from the UK and 10% from Canada. Ten percent (32 reviews) were judged to provide a comprehensive description of methods of involving stakeholders. Sixty-nine percent (22/32) personally invited people to be involved; 22% (7/32) advertised opportunities to the general population. Eighty-one percent (26/32) had between 1 and 20 face-to-face meetings, with 83% of these holding ≤ 4 meetings. Meetings lasted 1 h to ½ day. Nineteen percent (6/32) used a Delphi method, most often involving three electronic rounds. Details of ethical approval were reported by 10/32. Expenses were reported to be paid to people involved in 8/32 systematic reviews.

Discussion/conclusion

We identified a relatively large number (291) of papers reporting stakeholder involvement in systematic reviews, but the quality of reporting was generally very poor. Information from a subset of papers judged to provide the best descriptions of stakeholder involvement in systematic reviews provide examples of different ways in which stakeholders have been involved in systematic reviews. These examples arguably currently provide the best available information to inform and guide decisions around the planning of stakeholder involvement within future systematic reviews. This evidence has been used to develop online learning resources.

Systematic review registration

The protocol for this systematic review was published on 21 April 2017. Publication reference: Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R: Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Research Involvement and Engagement 2017, 3:9. https://0-doi-org.brum.beds.ac.uk/10.1186/s40900-017-0060-4.

Background

The concept of active involvement in research of people with a healthcare condition, their families, friends and carers, was founded on the principle that people affected by the condition have a moral right to contribute to decisions about what research is undertaken and in what way [1,2,3]. The active involvement of other stakeholders (meaning patients, the public, health professionals, health decision makers and funders) grew from a desire to address the lack of real-world relevance of research and to ensure more effective implementation of research findings into practice [4, 5]. It is now widely accepted in many parts of the world that the active involvement of many of these groups (that we collectively refer to as ‘stakeholders’) is beneficial to the quality, relevance and impact of health research [2, 3]. Accordingly, many funding bodies, including government and charities, now mandate that researchers actively involve patients and the public in their research, including systematic reviews [6,7,8,9], although there is evidence of international variation in the extent to which patients and the public are involved [10].

Systematic reviews aim to inform and support the delivery of evidence-based practice, by finding and bringing together, in an explicit and transparent way, all the research evidence that addresses a particular topic or healthcare question. Stakeholder involvement within systematic reviews has been proposed as a way to enhance the actual and perceived usefulness of synthesised research evidence, addressing barriers to the uptake of evidence into practice [11]. In this paper, we define (based on a number of published definitions, e.g. [1, 12, 13]) ‘active stakeholder involvement’ as the contribution of people who are not researchers throughout the process of production and dissemination of a systemic review, including the planning and conduct of an individual systematic review. While there are a number of examples of active stakeholder involvement in systematic reviews, the approaches to, and extent of, involvement have varied considerably [14,15,16] and synthesised evidence and resources to guide practice is lacking. As well as active involvement within individual systematic reviews, stakeholders may also get involved at the level of organisations which commission or carry out systematic reviews. A recent review explored examples of consumer involvement within organisations (such as Cochrane) that support production of systematic reviews [17], but evidence relating to relevant activities and roles of individual researchers and how they may involve stakeholders in their reviews remains scant [18].

As part of a wider project to provide guidance to researchers about how to involve stakeholders in systematic reviews [19], we undertook a mixed-method evidence synthesis, first completing a scoping review to create a broad map of evidence relating to stakeholder involvement in systematic reviews, followed by two contingent syntheses [20]. Here, we report the results of the scoping review. The aims of this paper are therefore to:

• 1. Document the evidence-base relating to stakeholder involvement in systematic reviews

• 2. Use this evidence to describe key features of how stakeholders have been involved in systematic reviews

Methods

Design

We carried out a scoping review, following a protocol [20]. We followed the methodological steps outlined by Arksey and O’Malley [21] and used an iterative team approach, with regular team meetings to discuss progress and reach consensus on next steps, to ensure clarity of purpose and balance between breadth and comprehensiveness of the review [21,22,23]. Protocol deviations, with justifications, are described in Additional file 1.

Search strategy

We implemented a stepwise approach [24] to promote efficient identification of up-to-date literature, balancing the expected large volume of literature with available time and resources. Details of this approach, including pre-agreed criteria and contingencies to inform decisions relating to the extent of the searches, have previously been described [20]; below we report the actual steps of searching and brief justification for these steps.

We used a comprehensive search strategy, adapted for each database (see Additional file 2). In step 1, we searched a comprehensive set of databases (CENTRAL (CDSR, DARE, HTA, Cochrane Methodology Register), Embase (Ovid), MEDLINE (Ovid), CINAHL (EBSCO), AMED, Joanna Briggs Database and ProQuest Dissertations and Theses (handsearched)), within a narrow time period (from 01 January 2014 to 09 April 2016). The aim of step 1 was to, in an efficient way, identify the databases most likely to include relevant papers. In step 2, we searched a more limited set of databases (Embase, MEDLINE, CINAHL and HTA) for a longer time period (01 January 2010 to 31 December 2013) with the aim of exploring whether there was justification for extending the search beyond 2010.

Searching and application of inclusion criteria was applied to each step prior to progression to the next step. For step 1, we noted the source database (or databases) of each identified record, and the databases from which the greatest number of included papers were identified. The results of these explorations were discussed and review team consensus reached on which databases to include in step 2. After step 2, the review team explored the publication dates of records meeting our inclusion criteria. The majority of papers meeting our inclusion criteria (63%) were published in either 2014 or 2015 (see Additional file 3). The sharp drop in numbers of included papers from 2014 to 2013 and relatively stable number of included papers between 2013 and 2010 were key factors in the team decision not to extend electronic searching to before 2010.

Additional sources we searched include the reference lists of recent relevant reports and reviews (e.g. [6, 17, 25]), the reference lists of all included studies and articles published in the journal Research Involvement and Engagement. To identify unpublished reports, we contacted authors of published papers and promoted this review via social media.

Selection criteria

Selection criteria for inclusion were purposefully wide. We included any paper, published or unpublished, regardless of study design, including commentaries, letters and expert opinion, which investigated, reported or discussed any aspect of stakeholder involvement in a systematic review. We anticipated that we would include (but would not be limited to) evidence such as published systematic reviews which reported involvement; reports of methods of involvement in an individual systematic review; studies quantitatively or qualitatively evaluating involvement in individual systematic reviews; and opinions, commentary and discussion relating to involvement in systematic reviews.

We excluded papers that focussed on stakeholder involvement in the generation of research priorities (unless they were specifically generating questions for a systematic review) and in both research more broadly, and guideline development, unless there was an explicit mention of involvement in systematic reviews. Systematic reviews that focussed on synthesising the evidence related to stakeholder involvement in primary research were also excluded. We excluded titles without abstracts and review protocols; this was a pragmatic decision made in light of the high volume of search results.

Definition of key terms

We used the following operational definitions, pre-stated in the protocol [20], to support the application of the selection criteria:

• Stakeholder—any person who would be a knowledge user of research but whose primary role is not directly in research. Potential stakeholders include a broad range of people, including those who are actual or potential recipients of health or social care, where this may include patients, carers and family members, or people interested in remaining healthy who are seeking information about a health condition or treatment for personal use [26]; members of organisations that represent people who use services; people with a professional role in health and social care; policy makers and managers. We documented the types of people involved within any evidence included in this review, highlighting where this included patients, carers and family members, and where this included other stakeholders only.

• Systematic review—a research process in which literature relevant to a stated question is identified and brought together (synthesised) using explicit methods [27], including reporting of inclusion/exclusion criteria, search methods and details of included studies. We accepted systematic reviews regardless of the type of evidence synthesised (i.e. quantitative, qualitative, mixed-methods) and the type of question addressed (e.g. intervention effectiveness, diagnostic test accuracy, patient experiences).

• Involvement in a systematic review—any role or contribution of stakeholders toward the development of a review protocol, completion of any of the stages of a systematic review or dissemination of the findings of a review.

Methods of applying selection criteria

One review author (PC) ran the search strategy and excluded any obviously irrelevant titles. Two reviewers (PC, AP) independently reviewed the abstracts and applied selection criteria to the first 500 records; agreement was explored and a full team discussion held to clarify the selection criteria. This clarification led to a number of post hoc exclusion criteria (described above under selection criteria and within Additional file 1). Subsequently, we agreed that two independent review authors (PC, AP) should review a further 500 records using the clarified criteria and that if agreement between independent reviewers was greater than 95% when using these refined criteria, then subsequent selection of papers would be performed by one reviewer only; this agreement was 97%, and therefore, one reviewer (AP) screened the remaining abstracts. The full papers from abstracts included after the screening process were considered at the data extraction and judgement stages (see below); if a paper was found not to meet the inclusion criteria at this stage, it was excluded.

Data extraction and synthesis

Data extraction

For all included papers, one reviewer (AP) extracted and categorised data into structured tables. Extracted data included bibliographic information, type of paper, stated aim, topic/focus of systematic review, study/review methodology, description of reported involvement, details of people involved, stage in review process at which people involved and any formal research methods used. Retrospective categorisation of data included focus of review and type of evidence synthesised (see Additional file 1, protocol deviations). Details of the operationalisation of these data extraction items are provided in Additional file 4.

Judgement of comprehensiveness of description

Our review aim was to describe key features of how stakeholders have been involved in systematic reviews; consequently, we were principally concerned with the comprehensiveness of the description of methods of involvement, rather than appraising the quality of the methods of the reviews. We devised a method for judging the comprehensiveness of the description of the method or approach to involvement, given that there are no standardised tools for such a task. Criteria for categorising the comprehensiveness of the description provided within papers were developed, adapted from Pollock [28]. Initially, two reviewers (AP, CS) assigned these criteria independently for a random sample of 20% of papers identified from step 1 of searching; this was 42 of 210 papers. There was agreement between independent reviewers for 57% (24/42) of the assessed sample. The agreement between reviewers, implications relating to disagreements and perceived risk of bias to the review results are reported in Additional file 5. Following discussion and clarification of criteria (see Additional file 5), it was agreed that one reviewer would assign judgements to the remaining papers, using the following criteria:

• ‘Green’—comprehensive description of one (or more) specific method or approach to the involvement of stakeholders in systematic reviews. Description sufficient to enable replication of methods

• ‘Amber’—a brief or partial description of one (or more) specific method or approach to the involvement of stakeholders in systematic reviews. Description sufficient to enable partial replication of methods

• ‘Red’—few details provided and/or inadequate description of the method or approach of involvement of stakeholders in systematic reviews. Description insufficient to enable any replication of methods

Detailed description of methods or approaches to involvement

Additional, more detailed, data extraction was performed for papers that were judged as ‘green’ for comprehensiveness of description. In addition to a narrative description of the methods or approaches to involvement, one reviewer (AP) extracted and tabulated the stated aim of involvement, number and characteristics of people involved, methods of recruitment, format of involvement (e.g. face-to-face meeting, telephone meeting, written consultation, online survey), amount of involvement (number of meetings, number of days involved), details of ethical approval and financial compensation given to stakeholders, evaluation of the involvement and tools used for reporting involvement.

Stakeholder involvement in this systematic review

One consumer (HG) and two consumer representatives (RM, CS) were members of the project and author team for this systematic review. All contributed to face-to-face discussions which led to the development of the review protocol, and read, commented on and had authorship of the published protocol. All contributed to project teleconferences throughout the review, particularly when making decisions relating to the stepwise search methods. Additionally, CS independently applied judgements of comprehensiveness to a sample of full papers. All three discussed the key findings of this review and contributed as authors to the final manuscript.

Results

Results of the search

We screened 12,908 titles and abstracts and applied selection criteria to 672 full papers. Three hundred sixty-nine of these 672 full papers were excluded: 118 as they were abstracts only, 18 as they were protocols, 16 as they were duplicates and 217 as they did not meet our inclusion criteria. Reasons that these 217 did not meet our inclusion criteria are listed in table of excluded studies (Additional file 6); main reasons for exclusion were that the paper was a systematic review but there was no involvement of people (approximately 30%), the paper did not describe or report a systematic review (approximately 25%) or the paper described involvement in research other than a systematic review (approximately 25%). This left 291 papers that met our criteria for inclusion in the scoping review (see Fig. 1).

Fig. 1

PRISMA flowchart

Characteristics of included papers

Details of the 291 included papers are provided in the table of included studies (Additional file 7). A brief summary is described below.

Type of paper

Thirty-one percent of included papers were published systematic reviews; 54% were reports of a guideline or recommendation in which a systematic review component was described; and 5% were papers specifically describing methods of involving stakeholders in a systematic review.

Stakeholders involved

Thirty percent of the included papers involved patients and/or carers within the systematic review process, while 41% involved other stakeholders (e.g. health professionals, academic experts, representatives of patient organisations) but not patients or their family members. In almost one third of the included papers (29%), it was not clear who the stakeholders involved in the review were and whether this included patients and/or carers.

Country

One third (31.6%) of papers were from the USA, one quarter (26.1%) from the UK and 10.0% from Canada. Of the remaining papers, 22.7% were from Australia, Netherlands, Germany, Italy, France or Spain, and 9.6% from a further 15 countries with between 1 and 4 papers each (see Table 1).

Table 1 Country in which stakeholder involvement took place/country of lead author

Stage of the review process

In almost half of the papers (47.8%), the stage of the review process at which stakeholders were involved was unclear. In just over one quarter (27.5%), stakeholders were involved in interpreting the results after the evidence had been synthesised. In around one fifth (22.3%), stakeholders were involved either throughout the whole review process or during one or more stages of review completion (see Table 2).

Table 2 Stage of the review process in which stakeholders were involved

Focus of the review

Seventy-one percent of the included systematic reviews were judged to be focussed on one of the International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10) categories (Table 3 and table of included studies (Additional file 7)). Most frequently (10%), this was ‘factors influencing health status and contact with health services’, where reviews covered topics such as the effectiveness or implementation of care pathways for specific population (e.g. paediatrics, geriatrics, emergency care). The specific diseases or health areas covered by the greatest numbers of reviews were mental and behavioural disorders (8.6%), neoplasms (6.9%), diseases of the musculoskeletal system and connective tissue (6.2%) and certain infectious and parasitic diseases (5.5%). Thirteen percent of the reviews which did not fit one of the ICD-10 categories were focussed on a specific intervention, most commonly medical or surgical interventions (8.6%) and public health interventions (5.2%). Ten percent of reviews were focussed on an area of research, rather than a specific health or disease area or intervention; more than half of these (55%, n = 29) were focussed on methods of stakeholder involvement or engagement, while the remainder focussed on other areas of research methods, such as methods of statistical tests within primary research. The remaining 7% were unable to be categorised within any of these groups and focussed on, for example, areas such as teaching, data protection and criminal justice.

Table 3 Focus of review

Comprehensiveness of description of method or approach to involvement

Table 4 shows the assigned judgements of the comprehensiveness of the description of the method or approach to involvement. Figure 2 illustrates the proportion of different types of paper which were judged to be ‘green’, ‘amber’ or ‘red’, when patients/carers were involved and at different stages in the review process.

Table 4 Comprehensiveness of description of method or approach to involvement

Fig. 2

Bubble plot illustrating proportion of papers in which patients/carers were included, the stage of the review process at which people were involved and the comprehensiveness of the description of the method of involvement

In total, 59% of the included papers were judged to provide few or inadequate details (‘red’), with only 10% judged to provide a comprehensive description of one, or more, method or approach to involvement (‘green’).

Detailed description of methods or approaches to involvement

The 30 papers which were judged as providing a comprehensive (‘green’) description of their methods or approaches to involvement included 14 ‘methods’ papers describing an experience of stakeholder involvement in one (or more) systematic review [25, 29,30,31,32,33,34,35,36,37,38,39,40,41]; 11 systematic reviews in which the stakeholder involvement was concurrently described [42,43,44,45,46,47,48,49,50,51,52]; 2 guidelines or clinical recommendations, in which the involvement in the systematic review component was described [53, 54]; and 1 paper which described the development of a tool to report stakeholder involvement [55]. Two of the papers each described two different systematic reviews [37, 55], meaning that there are a total of 32 systematic review described. Table 5 summarises the key characteristics of these 32 systematic reviews, and Table 6 summarises the data relating to stakeholder involvement and a brief narrative summary of key features is provided below.

Table 5 Key characteristics of ‘green’ systematic reviews

Table 6 Summary of data relating to stakeholder involvement in ‘green’ systematic reviews

Review aim/focus

Table 5 states the aim and focus of the 32 systematic review. Sixty-eight percent were focussed on one of the ICD-10 categories; with mental and behavioural disorders being the most common health topic (22%). Sixteen percent were focussed on a specific intervention rather than a disease area, most commonly on a public health intervention (12%). The remaining 16% were focussed on either research or another topic.

A majority of the reviews (56%) synthesised both qualitative and quantitative evidence, while 19% only included quantitative studies and 12.5% only included qualitative studies. The type of evidence included was unclear for 12.5%. Two of the reviews described using a ‘realist’ review methodology, and 2 were Cochrane reviews of randomised controlled trials.

People involved

Seventy-eight percent of the systematic reviews involved patients, carers or family members, while in one (3%), the people involved were peer support workers. In 19% of systematic reviews, the only people involved were professionals or academic experts, although one of these [56] aimed to recruit patient representatives, but failed to do so. Where there were face-to-face meetings, the number of stakeholders involved ranged from 2 to 27; where there were one-off events, often advertised as open to the general public, the numbers of stakeholders involved ranged from 15 to 81; where involvement did not require a face-to-face meeting, for example using an electronic Delphi or survey, the numbers invited ranged from 29 to 340 (see Table 6).

Geographical location (from which stakeholders were recruited)

The majority of the involvement occurred in the UK, with two thirds (66%) of papers describing UK-based activities. Of the remaining 34%, 2 recruited people from across Europe, 3 were carried out in Canada, 3 in the Netherlands and 1 in Australia, USA and Spain.

How people were recruited

For 69% (22/32) of the systematic reviews, people were personally invited to be involved. This involved personal invitations to known people (in 12/22; [29, 31, 39, 41,42,43,44, 48, 54, 55, 57]); personal invitations to an existing group or groups (in 6/22; [34, 35, 37, 46, 58]); or purposive sampling, using similar methods as sampling for qualitative research (in 4/22; [32, 38, 40, 56]). For a further 7/32 of the systematic review, involvement opportunities were advertised to the general population, often snowballing information out via target groups and organisations, and anyone who volunteered could get involved [25, 36, 47, 49, 50, 52, 53]. A combination of different recruitment strategies was used for 1 systematic review [33], and the method of recruitment was unclear for 3 systematic reviews [30, 45, 51].

Format of involvement

The format of involvement comprised direct, face-to-face interaction in 81% and an electronic Delphi method or survey in 19% of the systematic reviews. The face-to-face interaction was either in the format of a meeting (53%; [29, 31, 32, 34, 35, 38, 40,41,42,43,44, 46, 50, 52, 55, 58]), a larger workshop or public event (19%; [25, 36, 37, 47, 49]) or a combination of both of these (9%; [33, 45, 48]). In each of the 6/32 systematic reviews which used an electronic Delphi method, there was a specific and focussed aim of stakeholder involvement; in 4/32 [30, 53, 54, 56], this was broadly related to reaching consensus on factors, recommendations or statements arising from the results of the systematic review, and in 2/32 [39, 55], this was to reach consensus on the topic or focus of the systematic review.

Amount of involvement

Where there was direct face-to-face interaction, there could be between 1 and 20 meetings or events. The majority (83%) of the 24 reviews providing this information held 4 or less meetings (median 2 meetings), while one held 5 meetings plus 3 public workshops [45]. Three held multiple meetings (12, 15 and 20 respectively by [25, 46, 58]); in each of these three examples, the approach is described as ‘participatory’. Where reported, the length of face-to-face meetings varied from 1 h to ½ day. Generally, the Delphi approach involved three rounds of an electronic survey, although in one example after two rounds of Delphi voting there was a direct face-to-face consensus meeting [54].

Ethical approval

Details of ethical approval were reported for 31% of systematic reviews; for details, see Table 6. One paper reported that ethical approval was sought but not required [44]. No details relating to ethical approval were provided by the remaining 66% of papers.

Financial compensation

Expenses (such as travel, accommodation and care costs for family members) were reported to be paid to people involved in 25% of systematic reviews; in two, this was expenses only, while in six money or a voucher was provided in addition to expenses (see Table 6). No details relating to financial compensation are reported in the remaining 75% of systematic reviews.

Tools or method of reporting involvement

Thirty-four percent of the included papers had a clear method of reporting involvement. Four used some sort of tool, framework or checklist: Concannon et al. [42] developed and used a 7-item question for reporting stakeholder involvement in research, Liabo [46] used a framework for considering impact of involvement, Martin et al. [36] reported an evaluation based on reporting standards within Guidance for Reporting Involvement of Patients and the Public (GRIPP) checklist [59] and Saan et al. [55] developed and used a Tool for Recording and Accounting for Stakeholder Involvement (TRASI). Seven specifically reported activities which groups of stakeholders were involved in within a specific section of text, tables or supplementary files [30, 35, 37, 38, 47, 50]. Within the remaining 66% of systematic reviews, information about methods of involvement was not reported within a particular section, table or file, but was distributed throughout the paper.

Evaluation of the methods of involvement

None of the 32 studies carried out any formal evaluation of the impact of involving stakeholders; however, 28% collected data relating to the views and experiences of people involved. Of these, four used a questionnaire to elicit the views and experiences of stakeholders [29, 30, 36, 41]; three held a discussion with stakeholders in which they were encouraged to share or reflect on their experiences and perspectives [31, 33, 47]; and two had both a questionnaire and a discussion [38, 44]. In addition, Liabo [46] reported data arising from audio recordings and minutes of all meetings, Hyde et al. [34] described ‘impact’ within a table, while the reflections of the researchers on the process of involvement were discussed by others [35, 37, 40, 52].

Discussion

Key findings: evidence-base relating to stakeholder involvement in systematic reviews

We identified 291 papers describing stakeholder involvement in systematic reviews. Approximately two thirds of published examples describe UK activities, but we found examples from at least 24 countries. Reporting of who was involved, in what ways and at what stage in the review process was generally very poor, and the majority of the papers (59%) were judged to provide few details and/or an inadequate description of the method or approach of involving stakeholders. Thirty percent of systematic review teams clearly involved patients/carers, but in many cases (41%), the stakeholders involved health professionals, academic experts or representatives of patient organisations, but not patients or their family members.

We identified 30 papers, describing 32 systematic reviews, which we judged to have sufficiently comprehensive reporting to allow a more in-depth synthesis of methods or approaches to the involvement of stakeholders in systematic review. We have described key features of how stakeholders have been involved in systematic reviews, using data from these 32 examples. However, it was notable that, despite the selection of systematic reviews which were judged to provide a comprehensive description of one or more method of involvement, there was still inadequate (or absence of) reporting of a number of features in which we were interested. For example, the majority of papers did not provide any information relating to ethical approval or financial compensation to the stakeholders involved. A key contributing factor to the poor reporting relating to aspects of how stakeholders were involved may have been the lack of a tool or standardised method for reporting. On the few occasions where a particular tool has been used to support reporting of information relating to involvement, the tool has often been developed specifically by the systematic review authors. In many cases, the method of reporting comprises a written description of the activities in which stakeholders have been involved, but we found inconsistencies in the type of information presented and the location of this information within published papers.

Implications: methods of involving stakeholders in systematic reviews

The evidence which we have synthesised demonstrates that actively involving stakeholders within systematic reviews is feasible, and can be incorporated into a wide range of different types of systematic review. While there can be considerable variation in how stakeholders are involved, and the types of stakeholders who are involved, and there is currently an absence of evidence to directly inform choices for methods of stakeholder involvement within future reviews, a number of implications can be drawn from our synthesised evidence. In particular, evidence drawn from the 32 examples explored in this review can highlight some of the methodological decisions which may be made when planning stakeholder involvement in future reviews. These include:

• Will people directly affected by the healthcare topic addressed within the systematic review (i.e. individual patients, carers or family members) be involved? Will health professionals, academic experts or representatives from patient organisations be involved?

• How to find people to involve? Within our 32 examples, we found two key methods of recruiting stakeholders to be in systematic reviews; in the majority of our examples, there were personal invitations to known individuals or groups, but in some cases, recruitment occurred through advertising to the general population in order to get stakeholders to volunteer to be involved.

• How will people be involved? Within our 32 examples, two distinct methods of involving people in a systematic review were identified: (i) face-to-face meetings or events or (ii) electronic Delphi method. Where there were face-to-face meetings, these could be attended by invited participants only or could be an open event or workshop to which members of the public are invited to attend. Invited participants may only attend a small number (often between 1 and 4) of meetings during the course of a systematic review, but this may be much more where a participatory approach is used.

• How many stakeholders to involve? The current evidence base indicates that the number of stakeholders depends on the way in which they will be involved. Evidence from the 291 papers in our synthesis shows that 1 stakeholder may be a co-author on a systematic review, 2–10 stakeholders may be members of a steering group, 5–50 stakeholders may attend face-to-face meetings or focus groups and 20–400 stakeholders may participate in Delphi rounds or attend events or conferences.

• Use of research methods? Our examples highlighted that the following research methods have sometimes been incorporated into stakeholder involvement in systematic reviews: focus groups, interviews and a number of consensus decision-making techniques such as Delphi, Nominal Group Technique and voting/ranking processes.

Other issues to consider when planning stakeholder involvement in systematic reviews are whether ethical approval will be required and resources for payment of expenses and any other financial compensation or reward.

Although there is insufficient evidence to directly inform choices relating to who to involve and in what way, the findings arising from the 32 papers identified in this review have been used to produce, in collaboration with Cochrane Training, freely available online learning material and resources [60]. There have been many urgent calls for high-quality training materials, reporting guidelines and examples of best practice to support active stakeholder involvement and to enhance the relevance, usefulness and accessibility of systematic reviews [2, 16, 18, 33, 61]; the evidence from this review therefore can arguably currently play a key role in learning and support relating to active stakeholder involvement in systematic reviews.

Implications: reporting stakeholder involvement in systematic reviews

Recording and reporting of stakeholder involvement is important, both to ensure transparency in relation to the contributions and roles of different stakeholders within the review process and to contribute to the evidence base relating to this field. This scoping review highlights that the current reporting of involvement in systematic reviews is very poor and sometimes absent, and rarely provides a comprehensive description of who was involved and in what way. While there are a number of tools and frameworks which review authors could consider using (e.g. [36, 46, 55]), there is not currently any tool, guidance or recommendations specifically designed to support reporting of involvement within systematic reviews. Generic guidance relating to the reporting of stakeholder involvement in research has recently been updated (GRIPP2, [62]); however, this guidance has not been specifically tested for use with systematic reviews and has lacked international input during development. It is clear that there is an urgent need for improved reporting of involvement of stakeholders in systematic reviews. Such reporting should enhance the ability to develop evidence-based guidance around how to involve stakeholders in systematic reviews, and to explore and evaluate the impact of involvement.

Limitations

Identification of relevant systematic reviews and data extraction

It is unlikely we identified all relevant examples of stakeholder involvement in systematic reviews, as we adopted a pragmatic search approach aimed at efficiency within project time and resource constraints. This was compounded by poor reporting and inconsistent terminology in this area. We believe it is highly likely that there are many systematic reviews where stakeholders played a key role that our methods could not identify. Our decision to exclude titles without abstracts and review protocols at the study selection stage may have introduced publication bias into our results, with a bias toward inclusion of papers published in peer-reviewed journals. Only one review author extracted data from the included studies, and there is the potential that this may have introduced bias and errors in extraction. In an attempt to improve transparency and reduce data extraction errors, we copied and pasted data verbatim from included papers into an electronic data extraction sheet. This is reported in the table of included studies (Additional file 7).

Judgement to identify those with most comprehensive description

The agreement between independent reviewers when applying the ‘comprehensiveness’ judgement to a subset of papers indicated that there were disagreements on around 17% of ‘green’ categorisations. We did not have the time or resources to have independent judgement on a higher proportion of studies. We are therefore not confident that our subset includes all papers which may provide an adequate description of some parts of the methods of involving people in a review. However, as the aim of this phase was to identify and describe methods of involvement from examples of systematic reviews, the impact of potentially falsely including or excluding a paper from this subset was perceived to be low. We present the included ‘green’ papers as examples of systematic reviews in which there was involvement of stakeholders and take care to stress that these are examples rather than a comprehensive sample.

Our judgement of the comprehensiveness of the description of the methods was not a judgement of the quality of the involvement methods and only relates to the depth of the description of stakeholder involvement provided in the identified paper. Over half (54%) of the 291 included papers were reports of a guideline or recommendation, but only 2 of these were judged as ‘green’ for comprehensiveness of description. A potential explanation for this finding could be that stakeholder involvement is generally a core component of guideline development, but the primary focus of related journal publications is often the key clinical messages and implications, rather than the methods of the guideline, which are often fully described elsewhere. A judgement of ‘amber’ or ‘red’ for the comprehensiveness of the description of the method of involvement in the published paper is not an indication either that the quality of the methods was poor or that details of methods of stakeholder involvement are not available elsewhere.

Conclusion

This systematic review summarises evidence relating to the involvement of stakeholders in systematic reviews. We identified a relatively large number (291) of papers reporting stakeholder involvement in systematic reviews, but the quality of reporting was generally very poor. The level of reporting of involvement of stakeholders in systematic reviews, and the inconsistencies in which this is reported, must be improved so that guidance around how people can be involved in systematic reviews can be developed and the impact of involvement explored. This scoping review lends support to calls for high-quality training materials and examples of best practice to support active patient and public involvement and enhance the relevance, usefulness and accessibility of systematic reviews [2, 16, 18, 61, 63]. We identified a subset of 30 papers which we judged to provide a comprehensive description of stakeholder involvement in systematic reviews, and used these examples to summarise different ways in which stakeholders have been involved in systematic reviews. These examples arguably currently provide the best available information to inform and guide decisions around the planning of stakeholder involvement within future systematic reviews. This evidence has been used by Cochrane Training to develop online learning resources relating to how to involve people in systematic reviews [60], and has been used to develop a framework for describing stakeholder involvement in systematic reviews (Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Morris J, Watts C, Morley R: Development and application of a framework to describe how stakeholders have been involved in systematic reviews, submitted).