Supported local implementation of clinical guidelines in psychiatry: a two-year follow-up

Abstract

Background

The gap between evidence-based guidelines for clinical care and their use in medical settings is well recognized and widespread. Only a few implementation studies of psychiatric guidelines have been carried out, and there is a lack of studies on their long-term effects.

The aim of this study was to measure compliance to clinical guidelines for treatment of patients with depression and patients with suicidal behaviours, two years after an actively supported implementation.

Methods

Six psychiatric clinics in Stockholm, Sweden, participated in an implementation of the guidelines. The guidelines were actively implemented at four of them, and the other two only received the guidelines and served as controls. The implementation activities included local implementation teams, seminars, regular feedback, and academic outreach visits. Compliance to guidelines was measured using quality indicators derived from the guidelines. At baseline, measurements of quality indicators, part of the guidelines, were abstracted from medical records in order to analyze the gap between clinical guidelines and current practice. On the basis of this, a series of seminars was conducted to introduce the guidelines according to local needs. Local multidisciplinary teams were established to monitor the process. Data collection took place after 6, 12, and 24 months and a total of 2,165 patient records were included in the study.

Results

The documentation of the quality indicators improved from baseline in the four clinics with an active implementation, whereas there were no changes, or a decline, in the two control clinics. The increase was recorded at six months, and persisted over 12 and 24 months.

Conclusions

Compliance to the guidelines increased after active implementation and was sustained over the two-year follow-up. These results indicate that active local implementation of clinical guidelines involving clinicians can change behaviour and maintain compliance.

Background

Transferring research results into routine clinical practice is complicated; several studies have described implementation difficulties and the complexity of achieving performance change in health care [1, 2]. Single interventions are not effective solutions [3, 4]. Although knowledge about effective implementation strategies has increased their use, it has mostly only resulted in small to moderate improvements. Clinical practice guidelines are defined as 'systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances' [5]. Clinical guidelines can be used as tools [6–8], but a passive dissemination alone has rarely been effective in changing health care professionals' behaviour [1, 9]. Guidelines have modest influence on clinical practice unless they are successfully integrated into the clinical settings [10]. Guidelines aim to influence the treatment behaviour of practitioners. However, studies are needed to show that physicians exposed to guidelines provide better treatment [11].

There is a gap between evidence-based knowledge and current practice in many medical areas [9, 12], and how best to implement guidelines into routine care remains unclear [13]. Implementation of guidelines mostly entails complex interventions, and effective interventions are often elaborated in complicated procedures [14, 15]. Commonly evaluated multifaceted implementation strategies are audits and feedback, reminders, and educational outreach [2].

Successful implementation is not enough; there is also a need for continuous follow-up both of compliance to the guidelines and whether it is maintained over time. There are numerous studies showing that compliance returned to baseline after implementation of clinical guidelines [16]. So far, little has been accomplished regarding strategies for maintaining compliance. Objective measures are needed, e.g., quality indicators. Ideally, these should be derived from clinical guidelines that are based on scientific research or consensus among experts. These indicators should be measures of process and thus also measure quality of care [17]. Numerous indicators have been developed to evaluate and assess the care provided to patients with chronic physical illnesses [18], but there is lack of studies of care provided to patients with psychiatric disorders [19]. In addition, we have not found long-term follow-up studies in psychiatry on whether changes in practice after guidelines' implementation are sustained.

This study aimed to assess the effects at 12 and 24 months of an implementation intervention designed to improve documentation of quality indicators in accordance with clinical guidelines for treatment of depression and suicidal behaviour in patients at six clinics in Stockholm, Sweden.

Methods

Implementation of psychiatric guidelines in Stockholm

In Stockholm county, Sweden, a series of regional clinical guidelines regarding psychiatric disorders has been published and disseminated since 2002 [20, 21]. Providers and purchasers in collaboration with Stockholm Medical Advisory Board run the development work. The intention is to require the clinical guidelines to be implemented in all psychiatric clinics in the county in order to provide high quality care on equal terms for all of the county's citizens [20, 21]. A pilot study has been conducted on the implementation of clinical guidelines in the care of depression and suicide. Quality indicators derived from the clinical guidelines were used to study compliance. Our previous study showed that the indicators were feasible for audit and feedback as part of the implementation strategy, and a six-month follow-up showed favourable changes in clinical practice [22].

Settings and participating clinics

In the present study, clinical guidelines for assessment and treatment of depression and guidelines for assessment and treatment of patient with suicidal behaviours were implemented in six psychiatric clinics in Stockholm, Sweden. In Stockholm, treatment is provided almost exclusively by clinics in the public sector. All six psychiatric departments in Stockholm County were invited to implement the guidelines, and four departments decided to participate. Six general psychiatric clinics for adults were included; all were outpatient clinics in an urban area. The resources and organization were comparable. The two departments that declined participation did not differ from the ones that accepted participation in terms of organization of care, personnel resources, and population, as they had uniform contracts with the county council purchasing office. Six clinics in the four departments were randomly selected, and they were randomly assigned to an intervention group or a control group. Two of these clinics participated in implementing the clinical guidelines for depression, and two clinics in implementing the clinical guidelines for suicidal behaviours. Two clinics received the guidelines, but were not included in the intervention and acted as controls.

Implementation process at the intervention clinics

The study began in May 2003. The first author and an external psychiatrist supported the implementation process during the first six months. Local multidisciplinary teams, co-led by the external psychiatrist, including nurses, physicians, counsellors, and psychologists were established at each of the four active clinics. The teams were locally elected and participation in the local implementation work and meetings was voluntary. The first author presented the implementation study and the quality indicators for each team.

Implementation started with a baseline collection of quality indicators from medical records in order to analyze the gap between clinical guidelines and current local practice. On the basis of this, a series of seminars was conducted to introduce the guidelines according to the identified needs. The implementation teams learned to use strategies for improvements, e.g., following a cyclical process of change (plan-do-study-act model) approach [23], which was used to change local practices. Regular meetings then took place and the leaders of the teams promoted the value of implementation activities regarding patient sessions and clinical behaviour. At the meetings, all members of staff were involved in setting local goals for implementation based on the quality indicators. They were also encouraged to provide feedback and identify potential barriers and promoters to change. Feedback was given every month, based on the indicator scores, in order to ensure that improvements were gradually achieved and maintained. Local workshops at the clinics were conducted weekly during the study period, in which participants met to exchange useful approaches.

The active implementation strategies were based on organizational learning theory and previous knowledge of effective measures to change clinical practice. A learning organisation is described as a process of increasing the capacity for effective organisational action through knowledge and understanding [24]. Through the learning of individuals, the organisational routines are changed. One member of the research team (first author) performed site visits (academic outreach detailing) every month to the intervention clinics during the implementation period. Regular discussions of 'best practice' were held. Through facilitation, practitioners were helped to formulate and reflect on their practical knowledge and professional behaviour. Members of the local implementation teams participated twice in a regional network in order to enhance effective implementation strategies and experience during the study. The participants were encouraged to contact others in the network to exchange experiences and inspiration in the implementation work. During implementation, the adaptation of care defined by clinical guidelines was conducted by the implementation teams. A protocol for local use was developed to promote the adaptation of best practice, based on the clinical guidelines. A summary of the performed interventions is presented in Figure 1.

Figure 1

A summary of the performed implementation interventions.

Data collection

The data collection took place before the start of the study, and after 6, 12, and 24 months. Patient records from adult men and women with an ICD-10 or DSM-IV diagnosis of depression were eligible for inclusion in the study on the implementation of the clinical guidelines for depression. For the implementation of the clinical guidelines for suicide attempters, the inclusion criteria were patient records from adult men and women appraised at psychiatric emergency clinics after a suicide attempt. The first 120 medical records that fulfilled the inclusion criteria from specific dates were randomly selected from each clinic, identified through the administration system. This was repeated at 6, 12, and 24 months. For the data collection before implementation, 60 to 61 records were collected from each clinic. At the control clinics, 120 medical records were selected before implementation and 120 records at each data collection point during the follow-up period. Trained abstractors examined the medical records. Inter-rater reliability was assessed by a random replicate sample of 40 records. (Kappa 0.92 to 1.0). The study was approved by The Central Ethical Review Board at Karolinska Institutet.

Study population

A total of 2,165 patient records were included in the systematic assessment. The study of the implementation of the clinical guidelines for depression included 1,083 adult patients, mean age 36.3 years (SD 11.2) diagnosed with a depression according to ICD-10 or DSM-IV [25, 26]. There were no differences between the implementation and control clinics regarding gender and age distribution of the included patients. The study of the implementation of the clinical guidelines for suicidal behaviours included 1,082 adult patients, mean age 35.1 years (SD 14.7) At baseline, the mean age of the patients at the implementation clinics was lower (32.5 (12.2), versus 38.3(15.2), t = 2.8, p < 0.01) but there were no gender differences. At six months, there were no age or gender differences.

At 12 and 24 months there were more females and younger patients at the implementation clinics (74.6% versus 64.2%, Chi-square = 4.2, p < 0.1) (mean age 33.7(13.2) versus 40.4(19.0), t = 3.9, p < 0.001), (70.0% versus 52.5%, Chi-square = 10.7, p < 0.001)(mean age 33.3(13.4) versus 37.8(16.7), t = 2.7, p < 0.01).

Selection of quality indicators

Process indicators extracted from the clinical guidelines were used as indicators of compliance. A modified audit instrument by Gardulf and Nordström [27] was used to assess the presence of the quality indicators. Each indicator was rated on a assessment scale from zero to two. The presence of the quality indicators in the medical records was given a score from zero to two, (zero, recommended criteria to guidelines were not met; one, recommended criteria were partially met according to the definition; and two, a clear occurrence). In a subsequent analyses, we used the quality indicators a binary variables where one and two (i.e., partial or full adjustment to the recommendation) were compared to zero (i.e., no adjustment to the recommendation). As a sensitivity analysis, quality indicators were also categorised as two versus zero and one. We have found essentially similar results with this alternative approach (data not shown). For all indicators, higher scores were desirable and indicated a better compliance to the guidelines. The indicators also were summarised to a total score for each clinical guideline. The total score for the guidelines for treatment of depression was 22 points and 26 points for the guidelines for suicidal behaviour. Special recording forms were developed for the data collection. Quality indicators for implementation of the clinical guidelines for the care of persons affected by depression and clinical guidelines for suicidal patients are listed in Table 1.

Table 1 Quality indicators for evaluation of quality of care in depression treatment and care after a suicide attempt.

Statistical analysis

The data were analysed using STATA and SPSS for Windows, versions 10 and 16.0, respectively. Inter-rater reliability was analysed by calculating Cohen's Kappa. Differences regarding age and gender distribution of the included patient records at implementation and control clinics were analysed using chi-square test and T-tests. To address the nested structure of our data, we fitted random-effects logit models where we clustered patients within their health care providers using 'xtlogit' command in STATA [28]. Odds ratios were calculated for the dichotomized quality indicators comparing quality of care before (reference category) and after 6, 12, and 24 months, respectively.

Results

Compliance to the clinical guidelines for depression

Table 2 shows compliance at baseline, and 6, 12, and 24 months after implementation of clinical guidelines for depression, based on the quality indicators. The documentation of the quality indicators improved from the baseline in the four clinics where implementation was carried out, whereas there were no changes, or a decline, in the documentation of most quality indicators in those without implementation. For most of the quality indicators, the increase was recorded at six months and persisted over 12 and 24 months. Although, for a few quality indicators the 24-month follow-up audit showed a slight decrease compare to the measurement at 12 months.

Table 2 The compliance before, 6, 12, and 24 months after the implementation of clinical guidelines for depression in % (n).

The compliance for some indicators was low initially and after implementation showed considerable improvement, e.g., the compliance for structured suicide assessment rose from 40.2% (for a clear occurrence to guidelines) before implementation to at least 97.5% after (Table 2). Total score of the quality indicators for clinical guidelines for depression with 95% confidence interval are presented in Figure 2.

Figure 2

Total score of quality indicators for clinical guidelines for depression and suicide.

Compliance to the clinical guidelines for the management of suicide attempters

A similar pattern was seen in the documentation of the quality indicators in the clinics that implemented the clinical guidelines for suicide attempters. There was an increase of the documentation at six months, and the increase persisted over 12 and 24 months (Table 3).

Table 3 The compliance before, 6, 12, and 24 months after the implementation of clinical guidelines for suicidal behaviour in % (n).

Some indicators were more sensitive to change, e.g., structured suicide assessment for suicidal patients rose from 55.4% to 97.1% for a partial or clear occurrence of guidelines and specialist assessment rose from 50.4% to 91.7%. Figure 2 shows the total score of the quality indicators for clinical guidelines for suicidal behaviours with a 95% confidence interval.

Discussion

This paper describes an actively supported implementation of clinical guidelines in psychiatric settings and examined compliance before implementation and after 6, 12, and 24 months using quality indicators as measurements. The results showed that there was a consistent significant increase in the documentation of almost all of the quality indicators, that this occurred after a rather short period of time, and was sustained at almost the same level throughout the two-year study period. The increase was only observed in the intervention clinics and not at the clinics to which the guidelines were only disseminated. These findings imply that a systematic implementation approach gives sustainable change, at least over a two-year period, as documented by quality indicators. Our study describes the challenge implicit in real-world implementation aimed at improving the quality of care. The aim of all implementation is a change that remains after the support is withdrawn, and the results indicate that changes had taken place in the organization and structure of the care provided at the implementation clinics. In order to achieve these changes, an active implementation was needed and not just a dissemination of, or lecturing about, guidelines. This finding is in accord with earlier studies [11, 29, 30].

It could be assumed that the current clinical practice was close to recommended care as presented in the guidelines, because the latter were based on information easily available to all clinicians. However, we found that there were large gaps between current clinical practice and recommended practice according to guidelines, especially in the clinics where guidelines for suicidal patients were implemented. The implementation required complex changes in clinical practice, better collaboration, and changes in the organization of care.

There are several likely explanations for the observed improvements. First, local implementation teams with multidisciplinary members were established. This initiative was intended to develop collaboration for organizational learning of best practice and change of clinical practice. The teams were encouraged to involve all staff at the clinic in adapting the guidelines for local use. Using local teams facilitated collaborative partnerships, integrated knowledge, and action. Thus, the team members gained a deeper understanding of the context and challenges of the local health service.

Further, the interventions included audits and regular feedback, which helped the local teams to monitor the implementation. The aim was that the local teams would be able to choose the most important areas for intervention and to measure success in terms of improved compliance to the guidelines and outcomes. Previous studies have reported that this enhances learning and facilitate translation of insight to daily work [31, 32]. The organization should make use of the change process to implement changes of proven effectiveness regarding implementations strategies.

The feedback was based on quality indicators that were easy to use and showed a high inter-reliability. The indicators were all process indicators that had previously been the subject of discussion as to how to use them more effectively in mental health care, and they were not particularly controversial [33]. Furthermore, the changes are unlikely to be sustained if implementation does not include repeated measurements to access advancement and encourage modifications.

Another active strategy was that an outside researcher made regular visits to support the local teams. Moreover, all involved teams were part of a regional network that held regular meetings, because successful adoption of innovations often depends on interpersonal relationships within a system or an organization. An organization that supports knowledge sharing, and encourages observation and reflections is more successful at innovation and diffusion [34]. The network, as well as the visits, facilitated this. Although the teams worked locally, they were able to learn about organizational culture, implementation technique, and improvement models from colleagues in the regional network. Moreover, this supported the involved practitioners in analyzing, reflecting upon, and changing their own attitudes and behaviours. The goal was to transfer implementation technology into the participating organizations in order to continuously improve each organization's capacity for change.

Another critical issue for success of a diffusion of innovation strategy is leadership [35]. Leadership is described as an important factor in translating guidelines into clinical practice. Lack of support from leadership is identified as one of the greatest barriers [36]. According to Garside [37], leaders must continually show the desired direction of change, and support the staff in their new roles and new skills in a change of organisation or process. In the present study, the leadership was involved at an initial meeting at which the guidelines were presented. Because they had all volunteered to participate, they supported the implementation activities and created a culture in which the changes in clinical practice were possible.

Thus, a multifaceted intervention including a variety of active strategies was used [3], which previously has been reported to be more effective than passive strategies or just the use of feed-back or audit [38]. Shortell et al. [39] have suggested that five dimensions are needed for a successful implementation, i.e., process, strategic, cultural, technical, and structural. Our implementation program included all of these dimensions.

The standard of care is not the same as the quality of care. The quality of care provided by the clinician may be below, equal to, or even above the acceptable standard of care. Practice parameters are strategies for patient management, designed to assist health care professionals in clinical decision-making. The practice parameters describe the generally accepted practices, but are not intended to define a standard of care. The intentions with the quality indicators as presented in the clinical guidelines were to represent ideal practice. Thus, they could be used to measure deficiencies between current practice and ideal practice as defined in the guidelines, which would indicate an area for intervention. These practice parameters reflect the state of knowledge at the time of development of the guidelines, and most certainly need to be regularly updated.

Psychiatric disorders are of great importance in public health. Depression is now the fourth-leading cause of the global disease burden and the leading cause of disability worldwide. Depression is the most important risk factor for suicide, which is among the top three causes of death in young people ages 15 to 35 [40]. Depression seriously reduces the quality of life for individuals and their families, and often aggravates the outcome of other physical health problems. Because depression is highly treatable, and currently undertreated, it is an appropriate focus for improvement of the treatment by implementing available evidenced-based clinical guidelines. Guideline implementation studies in the care of psychiatric disorders are lacking, but a review by Weingartner of clinical guidelines in chronic medical diseases has stressed the importance of multifaceted interventions [41]. A comparable conclusion that multiple strategies seem to be most effective is presented in a systematic meta-review by Francke [16].

There were some limitations in the present study. Firstly, although both intervention and control clinics were randomly assigned, all had volunteered to participate, and therefore probably were more motivated to change. Secondly, given the fact that clinical practice change is a complex phenomenon dependent on local context, results from one particular setting can be generalised only with great caution [42].

Our study had a cluster design where patients were nested within their health care providers, and the health care providers were nested within their clinics. While the clustering at the provider level was properly addressed in our analyses, due to the low number of participating clinics it was not possible to fit a three-level model. Therefore, we could not investigate the possible role of clinic level covariates, and the lack of controlling for autocorrelation within clinics might inflate somewhat the standard error of our estimates.

Addressing local needs when implementing clinical guidelines is important in closing the gap between research and practice. The need to adapt implementation efforts to local circumstances has been shown to be valuable [43]. Adequate funding is needed to train the staff in the intervention techniques, establish protocols, and support evaluation of the outcome. Further research is needed on practical frameworks to facilitate the implementation of intervention in mental health care settings.

A large number of factors determine whether or not implementation will be successful and all factors cannot be addressed within one theory or model of change. Further studies are needed to examine our implementation approach with reference to theories about the implementation of change. The strength of the present study is that it is, to our knowledge, the first one to assess the long-term effects of implementation of psychiatric guidelines.

Conclusions

This study suggested that the compliance to clinical guidelines, for treatment of depression and suicidal behaviour, was implemented and sustained over a two-year period after an active implementation. Quality indicators were helpful tools in the implementation process as well as in the evaluation. Thus, supported local implementation based on local organisation theory may be a strategy for narrowing the gap between evidence-based care and current practice.