Intervention Scalability Assessment Tool: A decision support tool for health policy makers and implementers

Abstract

Background

Promising health interventions tested in pilot studies will only achieve population-wide impact if they are implemented at scale across communities and health systems. Scaling up effective health interventions is vital as not doing so denies the community the most effective services and programmes. However, there remains a paucity of practical tools to assess the suitability of health interventions for scale-up. The Intervention Scalability Assessment Tool (ISAT) was developed to support policy-makers and practitioners to make systematic assessments of the suitability of health interventions for scale-up.

Methods

The ISAT was developed over three stages; the first stage involved a literature review to identify similar tools and frameworks that could be used to guide scalability assessments, and expert input to develop draft ISAT content. In the second stage, the draft ISAT tool was tested with end users. The third stage involved revising and re-testing the ISAT with end users to further refine the language and structure of the final ISAT.

Results

A variety of information and sources of evidence should be used to complete the ISAT. The ISAT consists of three parts. Part A: ‘setting the scene’ requires consideration of the context in which the intervention is being considered for scale-up and consists of five domains, as follows: (1) the problem; (2) the intervention; (3) strategic/political context; (4) evidence of effectiveness; and (5) intervention costs and benefits. Part B asks users to assess the potential implementation and scale-up requirements within five domains, namely (1) fidelity and adaptation; (2) reach and acceptability; (3) delivery setting and workforce; (4) implementation infrastructure; and (5) sustainability. Part C generates a graphical representation of the strengths and weaknesses of the readiness of the proposed intervention for scale-up. Users are also prompted for a recommendation as to whether the intervention (1) is recommended for scale-up, (2) is promising but needs further information before scaling up, or (3) does not yet merit scale-up.

Conclusion

The ISAT fills an important gap in applied scalability assessment and can become a critical decision support tool for policy-makers and practitioners when selecting health interventions for scale-up. Although the ISAT is designed to be a health policy and practitioner tool, it can also be used by researchers in the design of research to fill important evidence gaps.

Background

In order to achieve population-wide benefits and foster sustainable policy and programme development, health interventions found effective within controlled or research settings should be scaled up [1, 2]. Here, we refer to the process of ‘scale-up’ or ‘scaling up’ as “deliberate efforts to increase the impact of successfully tested health interventions so as to benefit more people and to foster policy and program development on a lasting basis” [2]. However, there have been few documented examples of efficacious population health interventions being scaled up successfully in developed countries [3,4,5,6]. A contributing factor is the lack of pragmatic studies demonstrating how pilot interventions can be disseminated in real-world settings, as research has predominantly focused on describing disease risk patterns [7] or intervention efficacy testing, rather than disseminating interventions across systems [5, 8, 9]. Other factors include lack of knowledge, skills and capacity among policy-makers and practitioners to determine the suitability of interventions for scale-up [10], and the likelihood that political and resourcing factors are often more powerful influences on scale-up decisions than whether interventions are evidence based [8]. Hence, it is important to provide better support to policy-makers and practitioners to more readily assess the suitability of interventions for scale-up and their scalability within a specific context.

In this paper, we define scalability as “the ability of a health intervention shown to be efficacious on a small scale and/or under controlled conditions to be expanded under real world conditions to reach a greater proportion of the eligible population while retaining effectiveness …” [1]. Assessing scalability has been identified as a fundamental step in any scaling up process [2, 11,12,13], as it helps to avoid unnecessary expenditure of resources and efforts to scale up unsuitable interventions [14]. Furthermore, assessing scalability generally requires an assessment of a range of considerations, including feasibility, acceptability, costs, sustainability and, most importantly, adaptability (i.e. to suit the needs of the context in which it is to be scaled up), which are often difficult to assess [1]. While a growing number of frameworks and guides offer step-by-step processes for scaling up evidence-based interventions and identifying factors, including scalability, that should be considered throughout the scaling up process [15,16,17], these guides do not offer practical tools that policy-makers can use to conduct structured scalability assessments. Where scalability tools do exist, such as in Cooley et al.’s [11, 18] Scaling Up Management Framework, the scalability checklist is brief and does not provide a mechanism for evidence gathering or a process for the comprehensive and systematic assessment of scalability. Moreover, the Scaling Up Management Framework checklist was developed primarily for use in low- to middle-income country (LMIC) contexts and thus may have less utility in high-income countries.

The aim of this study was to develop an Intervention Scalability Assessment Tool (ISAT) that enables policy-makers and practitioners to make systematic assessments of the suitability of health interventions for population scale-up within high-income country health and community settings.

Methods

The ISAT was developed in three stages; the first stage involved a literature review and expert consultation to identify existing scalability tools and the potential domains to be covered by the initial version of the ISAT. The second stage involved testing the initial version of the ISAT, with five end users and developing a second draft based on respondent feedback. The third stage involved testing the second version with 24 end users and using feedback to develop the final ISAT. Figure 1 summarises the key steps in the development of the ISAT. More detail on the methods employed in each stage follows.

Fig. 1

Summary of the research process

Stage 1: Development of the first version of the ISAT

A literature review was conducted to identify frameworks, guides, checklists or tools associated with the scalability or scaling up of health interventions that could be used to inform the development of the ISAT. The review was conducted in two phases. In Phase 1, a keyword search of studies published in English between 1990 and 2017 was performed using MEDLINE and SCOPUS. The search terms were (“scalability” OR “scale up” OR “upscale” OR “up-scale”) AND (“framework” OR “guide” OR “checklist” OR “tool” OR “readiness” OR “assessment”).

The logic behind including scale-up frameworks in the review was that ‘scalability assessment’ is an initial step in a number of such frameworks [1, 11, 13]. The search strategy was developed by AM and abstracts were retrieved and assessed for relevance to the study by KL. In Phase 2, among abstracts deemed relevant in Phase 1, full papers were retrieved and assessed against the following review inclusion criteria:

• Peer reviewed

• Published in English from 1990 to 2017

• Described frameworks, guides, checklists or tools associated with scalability considerations or scale-up of health interventions

For the purposes of the review, ‘frameworks or guides/guidelines’ were included if they provided structured approaches or step-by-step guides by which scale-up or scalability could be organised or assessed. ‘Checklists’, ‘tools’ or ‘assessments’ were included if they provided a list of features or considerations with respect to making decisions regarding scalability or scale-up.

Papers were excluded if they described:

• The concept of scalability without providing a framework or checklist

• Scale-up of information technology systems not related to health interventions

• Scale-up and evaluation of specific interventions

• Study protocols for potential scale-up

• Medical testing procedures

• Statistical modelling without frameworks, guidelines or tools to assist decision-makers to assess the scalability of health interventions

• Facilitators and/or barriers to scale-up within specific interventions or general experiences of scale up that did not provide a framework or checklist for guidance

• New scale-up concepts that did not include guidance on how they were to be applied

To supplement the peer-reviewed scientific literature, an open keyword search in the Google search engine was also conducted using the same variations of the above search terms. As recommended by the Canadian Agency for Drugs and Technologies in Health [20] for systematic searches of grey literature, the top 50 hits were reviewed. The reference lists of all included papers were further scanned for potentially relevant tools or articles.

Finally, a convenience sample of Australian experts (n = 3) in scaling up health interventions were also consulted to identify sources in the grey and peer-reviewed literature of potential relevance to the task of developing the draft ISAT. Experts were selected for their knowledge of the scale-up literature, implementation frameworks and experience with scale-up processes.

The relevant information from the results of the peer-reviewed and grey literature review was extracted and integrated to map existing frameworks, guides, checklists and tools used for scaling up health interventions and scalability assessment across key characteristics, as shown in Table 1. Table 1 describes the terms and categories used to distinguish the features between the frameworks, guides, checklists and tools outlined in Table 3.

Table 1 Definitions of categories

AM produced an initial draft of the ISAT in light of the literature and expert opinion. The domains identified in this initial draft were drawn from those described in existing scalability checklists and scale-up frameworks. This initial draft was then circulated among the study investigators for over several rounds and their feedback was incorporated into the draft of the ISAT tested in the first round of end-user interviews.

Stage 2: End-user interviews (Round 1) and testing

To test the draft versions of the ISAT, interviews were conducted with a range of Australian policy-makers, implementation science academics and clinicians (hereafter known as ‘end users’) with experience in scaling up health interventions. Relevant end users with experience in scaling up public health interventions were initially identified by study investigators and followed by a passive snowballing recruitment method to obtain additional informants. Interviews with policy-makers and practitioners were conducted over two rounds; the first round was conducted between December 2017 and January 2018 and, following further revisions to the ISAT, a second round was conducted between May and September 2018. A total of 34 end users were invited to participate in the process, from which 29 interviews were conducted (85% response rate). The remaining end users (n = 5) were not able to be contacted or did not respond. The majority of interviews (n = 25) were conducted over the telephone, and a minority (n = 4) were conducted face to face. Different end users participated in each round of interviews.

For both interview rounds, topic guides were developed and questions were generated with a view to understanding the processes that policy-makers use to make decisions on scaling up health interventions and the difficulties encountered in that process as well as to obtain feedback on the content and structure of ISAT drafts. Ethics approval was obtained from the University of Sydney’s Human Research Ethics Committee (2017/828). A summary of the topics covered in the two rounds of end-user testing can be found in Table 2 and the full interview guides are provided in Additional file 1.

Table 2 Summary of topics covering end-user interview guides

KL and KC conducted interviews, with KL acting as the lead interviewer. The interviews were audio recorded and transcribed verbatim by an external transcription service (www.rev.com). All end users were invited to participate via email and provided their verbal and signed consent. All interview transcripts were de-identified for the purpose of analysis and reporting. Thematic analysis [32] was used to identify themes across the interview responses and were used to reshape each iteration of the ISAT. The analysis was conducted by KL and findings were discussed with all members of the study investigative team and used as part of the revision process. Between each round of interviews, the responses obtained from end users were collated and thematically coded and used to revise the ISAT.

Stage 3: End-user interviews (Round 2) and development of the final ISAT

The last stage of the process included the thematic analysis and coding of end-user responses from the second round of interviews to produce the final ISAT.

Results

Literature review

The initial search in MEDLINE and SCOPUS databases with the keywords and keyword combinations yielded 2769 abstracts and searches of the grey literature identified a further 4 documents, making a total of 2773 documents. Following exclusion of duplicate records (n = 134), of the 2639 paper abstracts and documents reviewed for relevance against the review criteria, 2554 were excluded.

The full text versions of remaining papers and reports (n = 85) were retrieved and reviewed against the inclusion criteria for Phase 2 (Fig. 2). A total of 28 papers described scaled up interventions and/or evaluations of scaled up interventions. A further 19 papers provided insights into facilitators and barriers to scale-up, but generally specific to their intervention and/or context. There were a small number of papers either generating new scale-up concepts or providing opinions on scale-up in general (n = 15). Finally, 15 papers/reports were noted as describing scale-up frameworks, tools and checklists, and were included in the final review. The literature search is reported according to the Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) statement (www.prisma-statement.org.au).

Fig. 2

Literature search PRISMA flow chart

Characteristics of existing frameworks, guides, checklists and tools used to scale-up health interventions

The literature review revealed a number of existing frameworks, guides, checklists and tools (hereafter known as ‘frameworks’) to guide the scaling up of health interventions (Table 3). Eleven of the 15 frameworks arose out of (or were developed specifically for) the global health and/or LMIC context. These articles had a primary focus on communicable diseases such as HIV/AIDS or general LMIC health promotion programmes, including maternal and child health improvement programmes. Similarly, 70 abstracts were excluded following the full text review Phase 2 as they only provided information on facilitators, barriers and case studies of successfully scaled up programmes from the global health or LMIC contexts.

Table 3 Existing frameworks, checklists and tools used for scaling up health interventions and scalability assessment

All of the frameworks and scaling up processes were developed using existing literature (theories and frameworks) and/or qualitative interviews with those who were part of a scale-up process or case studies. Of the 15 individual papers shown in Table 3, nine were primarily focussed on providing a framework for the scale-up process (i.e. dissemination), while two were focused on implementation. The remaining four either focused on what we have termed pre-scale-up (i.e. scalability considerations exclusively [1]) or as an important part of the scale-up process [11, 18, 22].

Of the ten frameworks and checklists providing steps for scale-up, five [1, 11, 13, 18, 22] specifically mentioned the assessment of scalability although guidance on assessing scalability was only provided by one author through two iterations [11, 18] of a scalability checklist (Table 3). This one-page checklist (called the ‘Scaling Up Management Framework Scalability checklist’) described seven categories for scalability and, within those categories, provided a series of questions to facilitate a decision of scalability across a three-point scale; users were not prompted to provide additional supporting text or evidence. The remaining three papers described scalability [1, 13, 22] and outline some of the key factors for consideration but only provided minimal or no guidance on how to perform a scalability assessment.

Table 3 summarises existing frameworks, checklists and tools used for scaling up health interventions and scalability assessment.

Table 4 describes key scalability concepts covered in existing scalability frameworks and checklists. Concepts covered in all of the existing scalability frameworks and checklists identified in the review included contextual considerations, evidence of effectiveness, scale-up and implementation considerations, workforce considerations, and costs of scale-up. Other common scalability concepts identified included the importance of problem definition, intervention adaptability and delivery system considerations. Only three of the five scalability frameworks and checklists covered the concept of intervention reach and acceptability.

Table 4 Scalability concepts covered in existing scalability frameworks and checklists

Table 5 summarises the key functions of the existing scalability frameworks and checklists. Of note, only two of the five existing scalability frameworks or tools provided structured scalability assessments [11, 18] and none of the assessment tools provided a process for evidence gathering across different scalability domains.

Table 5 Key functions of existing scalability frameworks and checklists

Table 6 Characteristics of the end users interviewed

Thematic analysis of interviews with end users

Respondent characteristics and scaling up experience

The end users interviewed (n = 29) were from all Australian states and territories and represented a number of different organisation types and roles (Tables 6, 4). All indicated that they had personally either been a part of, or had led, the scale-up of health interventions.

End users mainly identified as mid-level policy-makers and came from state and/or local-based health organisations. Those identifying as Senior Executives indicated that they had been and/or are currently part of the decision-making capacity within their organisations on funding and scale-up of interventions. Data from interviews with respondents are described below under the themes of importance of decision support tools, when and how to use the ISAT, perceptions on the likely process required to complete the ISAT, additions to the ISAT and potential limitations. This thematic analysis informed the development of ISAT and is followed by a description of the final ISAT.

Importance of decision support tools for policy and practice

The concept of an assessment tool that aims to facilitate structured thinking and consideration of potential implementation issues when scaling up promising interventions was widely supported amongst interviewed end users. From a decision-maker perspective, it was reported that having well thought-out considerations generated by completing the ISAT would prompt more detailed discussion of the challenges that might be faced when implementing an intervention at scale and, by doing so, would help improve implementation-related decision-making.

“I think this is a tool that would help people to craft well thought through arguments as to why they think a scaled up version of a program would be of benefit, and the kinds of impacts and investment it … Well, the investment that you’d need and the delivery system that you’d need, and then the kind of impacts that you’d see at a population level, they’d be the sorts of things that then become important in even considering an idea.” (P04)

Others expressed that a tool like the ISAT could be included as part of funding application processes as it would add rigour, consistency and, potentially, accountability to funding assessment processes. Being able to standardise the application of the scalability assessment tool to funding decision-making processes would also be useful where there are competing claims on budgets with multiple interventions for consideration.

When/how to use a decision support tool like the ISAT

End-user interviewees reported that there was value in using the ISAT beyond assisting decision-making on whether to fund interventions for scale-up, and for informing decisions about interventions that are currently being scaled up or delivered at scale (i.e. whether to continue, expand, scale back or terminate these activities). Other potential uses identified for the ISAT were as a tool to advocate for further funding or support for a promising intervention, as part of the strategic planning process in determining promising interventions for further testing and/or potential scale-up, and as a mechanism for documenting information and processes of the intervention and its subsequent scale-up to capture key learnings for quality improvement activities.

Perceptions on the likely process required to complete the ISAT

Despite not having piloted the ISAT on a real-life intervention, a number of end users suggested that the ISAT would be best completed as a group process since the relevant data and information for answering the questions is unlikely to be found in any one location or from one individual alone. Many indicated that having multiple viewpoints to inform the final assessment process would be valuable in facilitating a more balanced and comprehensive assessment.

Suggested additions to the ISAT

While not listed in detail in this paper, there were numerous suggestions for additions, exclusions and modifications to the content of the ISAT through both rounds of interviews. One of the key additions included the need for greater emphasis on the consideration of sustainability of interventions post scale-up. Respondents reported that, in their experience of scaling up, sustainability (particularly financial sustainability) was often not considered beyond the initial funding period. Using the ISAT in the planning stages where sustainability is addressed could help increase the likelihood of longer-term sustainability when it is implemented.

The overall comparison across the key domains of the tool was added to the tool following the initial round of user feedback. Respondents reported that the resultant summative visualisation of the strengths and weaknesses of interventions considered for scale-up, using a mix research evidence, expert opinion, practitioner knowledge and contextual information, was very useful for decision-making.

Potential limitations of the ISAT

A number of limitations of the ISAT were identified through the interviews with end users, the first of which was that, while the ISAT could provide guidance on the scalability of efficacious interventions, its ability to provide guidance for interventions where the evidence base is not as strong or yet to be tested was challenging for some. Examples of this included interventions that were still in a pilot testing stage.

Secondly, it was noted that, potentially, those with less experience with the concepts and requirements of scaling interventions might find completing the ISAT challenging. There is a certain level of implied knowledge and skillset in order to complete the ISAT, leading to the observation that the ISAT would be best completed as part of a group process where different views and expertise would contribute to a balanced and comprehensive assessment.

The ISAT

The ISAT (Additional file 2) consists of three key parts — Part A: setting the scene, Part B: intervention implementation planning, and Part C: summary of scalability assessment — as outlined below.

Part A: setting the scene

The purpose of this section is to outline the context for which the intervention is being considered for scale-up and consists of five individual domains, as outlined in Table 7.

Table 7 ISAT: PART A domains and objectives

Part B: intervention implementation planning

Part B considers the potential implementation and scale-up requirements of the intervention within five domains. The questions are designed to promote early thinking about potential implementation issues that would contribute to an intervention’s potential scalability. The information generated through this section can provide the basis for a detailed scale-up or implementation plan (Table 8).

Table 8 ISAT: Part B domains and objectives

Part C: summary of scalability assessment

At the end of each domain there are several questions, rated on a scale from 0 to 3, representing the readiness assessment for that domain. The purpose of these questions is to identify strengths and weaknesses for comparison across domains. Part C amalgamates the results from Parts A and B, including the summation of the assessment scores. A visual representation of the results of assessments for each domain in the form of a spider web plot may be generated to summarise the strengths and weaknesses of the proposed intervention in light of the scalability criteria and facilitate cross-domain comparisons (Fig. 3).

Fig. 3

Example of ISAT spider web plot

The final question prompts a recommendation about whether the intervention should be recommended for scale-up, whether the intervention is promising but likely to require further information or planning before scaling up, or whether it does not merit scale-up.

Interpreting the ISAT findings

The questions in the ISAT are wide ranging and their intent is to stimulate thinking and promote the active consideration of factors that are important when assessing scalability. It is important to note that there is no ‘total’ or summary score that can be derived from combining the scores on the individual domains. The omission of such a total score is deliberate in recognition that not all the domains are necessarily equally important across all contexts and scenarios. Low scores in some domains may be acceptable in some situations, but not in others, depending on the perceived importance of the domain in the context of intervention of scale-up. Further, it may not be possible to answer all questions posed in the ISAT, but the absence of information is also informative. Visual representation of the scores across each domain serves to highlight areas in which evidence may need to be strengthened. It also provides a starting point through a structured process to facilitate discussions on the potential scalability and/or readiness of the intervention in question with a variety of decision-makers and other stakeholders.

Discussion

While a variety of frameworks to guide scale-up processes exist in the literature, this review identified a major gap in the area of scalability assessment, particularly in high-income country contexts. It is worth noting that, while two scalability checklists were found in the literature [11, 18], both were from the same author and were simple, one-page checklists. These checklists ask users to tick responses to questions under different scalability domains but does not collect evidence under each domain to enable the graded assessment required for a comprehensive scalability assessment.

The three-part ISAT guides users through the known factors that affect the success of scaling up health interventions, encouraging evidence-based decision-making and reflection on resources required and potential threats to sustainability. The core domains contained in the ISAT were largely derived from the scale-up literature and refined based on feedback through interviews with end users. Key domains in Part A, such as defining the problem, intervention characteristics and contextual factors, are all scalability considerations reported by Milat et al. [1] and Spicer et al. [22]. While Part B focuses on implementation, including domains such as the potential reach, adoption, acceptability of the inetrevention along with potential resource requirements. These domains are consistent with key success factors for scale-up and/or effective implementation identified in the literature [1, 8, 11, 12, 18, 27, 28]. Interestingly, only three [1, 13, 23] of the five existing scalability frameworks and checklists identified in the review covered the concepts of intervention reach and acceptability. These were thought to be vital scalability concepts by respondents in this study and, as such, they feature prominently in the ISAT. The potential adaptability as well as future sustainability of the intervention featured in this section are highlighted in the literature as important considerations in effective scale-up [25, 33,34,35]. The final section of the ISAT (Part C) provides a summative assessment that highlights the strengths and weaknesses across domains to inform a final decision on whether to scale up an intervention or not. This feature has not appeared in any existing tools identified in the literature. The convergence between the literature, the selected domains and the last round of end-user testing was encouraging, indicating that there were no major omissions identified in the final version of ISAT.

The end users consulted for the study identified a number of important uses for the ISAT, including as a decision support tool, as an advocacy tool, and for use in strategic planning and resource allocation processes. Many of those interviewed noted that political and/or strategic and organisational context and support along with availability of funds, often take priority over evidence of efficacy in scale-up decision-making. However, respondents also believed that a tool like the ISAT would be very useful in bringing to light the evidence base for a particular intervention and in inserting that evidence into decision-making processes through a structured and transparent process. It was noted by respondents that the process of completing the ISAT (especially as a team) provides a structured opportunity through which policy teams can interact with and consider evidence as part of the decision-making process.

We recommend that the tool is completed by a group of stakeholders as it will often be difficult for a single individual to gather evidence for all the domains in isolation. The team assembled to complete the ISAT will be context specific but would generally bring together a range of expertise in research, programme planning, implementation and practice. Teams could involve policy-makers, researchers, implementers and context-specific practice experts. For example, research expertise will facilitate assessments regarding the strength of the evidence, while practitioner/policy-maker experience will be critical in identifying important contextual factors that could affect implementation and the scale-up process.

Utilising a tool like the ISAT as part of the scaling up decision-making process can help policy-makers and practitioners distinguish between effective and implementable interventions. For example, some interventions may not be scalable to the population level despite showing effectiveness in controlled settings due to deficiencies in other important domains such as cost, workforce and sustainability. Completing the ISAT can also enable greater consistency when comparing across interventions competing for the same funds. The process of completing the ISAT itself may also promote learning in how to design programmes for at-scale implementation.

Broadly, the tool has been designed with high-income countries and complex population health interventions in mind as this was identified as a gap in the literature. However, this does not preclude its use in LMICs and other contexts or settings such as human services. The ISAT could also be used by research funding agencies to assess whether they should fund proposals to scale up health interventions and it could be used by researchers in the design of research studies to fill important evidence gaps.

It is important to acknowledge that all the information that would ideally be required to assess the scalability of an intervention may not be available at the time the assessment is made. For example, it might not be possible to accurately determine how large the effect size at a population level needs to be to achieve a population health gain or how much the programme can be changed (to reduce cost or suit different contexts) while still retaining fidelity and outcomes. Where there are gaps in the available evidence, decision-makers may need to consider information from other sources such as expert advice, practice-based knowledge or parallel evidence from similar programmatic interventions in other fields. Where no information is available, a judgement is required about how important the missing information is, whether any gaps can be addressed during implementation, or whether further research is required before scaling up can be recommended. The ISAT provides a systematic way of identifying these gaps and assessing their relative importance.

End users reported that the ISAT could provide guidance on the scalability of efficacious interventions; however, some respondents felt that its ability to provide guidance on the scalability of interventions with weak evidence or those yet to be tested was problematic. We argue that only efficacious/effective interventions should be scaled. Nevertheless, our interviews demonstrated that the reality is often different, with a recent review of scaling up pathways of chronic disease prevention interventions showing that 15% of scaled up programmes identified were based on no discernible evidence of intervention efficacy/effectiveness [36]. In any scale-up process there will likely always be gaps in evidence and, as stated above, it is reasonable to proceed with these gaps in less critical areas. One exception to this rule is intervention efficacy/effectiveness; the risk of scaling up interventions without such evidence is great and may result in the scale-up of programmes that do not work and can divert scarce resources away from potentially effective interventions. This is an important risk that must be managed by decision-makers as it can result in denying or delaying community access to effective services and, ultimately, to the superior health outcomes that these services can provide.

The concepts and content contained in the tool are comprehensive. However, it is not possible for any tool to include every aspect of scale-up nor is this always desirable for practical reasons. It is vital that any tool to be used by decision-makers, like the ISAT, should not be overly onerous to complete. While the tool encompasses elements identified in the literature as key considerations of scalability [1, 11, 13, 18, 22], it is acknowledged that gaps may still exist.

Completing this tool does not negate the need for a comprehensive scale-up plan and/or implementation plan to be developed subsequent to a decision to scale up. Importantly, the information collected by the ISAT could also be used to inform the development of the forthcoming scale-up and/or implementation plan. Further, as some of the end users indicated in their interviews, having the right level of skills mix and staff to complete the tool as a group would be useful but potentially challenging in terms of making it a reality.

Limitations of the study and further research

This study engaged a small number of scale-up and implementation experts, policy-makers and practitioners, all from Australia. While a larger sample or different respondents may have generated some differing views, the consistency and detail of responses to the semi-structured interviews as well as the substantial experience of the respondents, lend confidence to the external validity of results. Further, the purposeful approach taken in respondent selection, the high response rate (85%) and the comprehensive engagement from all respondents implies that the aforementioned limitations were likely to have been minimised.

The ISAT has been designed to be a practical decision support tool and has been turned into an Australian Prevention Partnership Centre guide for use by policy-makers and practitioners. A logical next step will be to test the usefulness and applicability of the ISAT in real-world intervention policy decision-making processes. We encourage its practical application to policy and programme decision-making in numerous contexts within health and social policy and we intend to conduct further testing of the ISAT in real-world scale-up decision-making processes and to share these as case studies with the field.

Conclusion

The ISAT fills an important gap in the literature by providing a tool for policy-makers and practitioners to make systematic assessments of the suitability of health interventions for scale-up. The ISAT is designed to stimulate thinking and promote active consideration of the factors that have been shown to be important when assessing scalability. Future research should test the ISAT using real-world case studies. The authors encourage researchers and policy-makers to publish these efforts and to share learnings in the field.