1 Introduction

The adverse drug reaction (ADR) causality assessment is a routine procedure in Pharmacovigilance[1], because it allows assessing drug safety parameters and the relationship and likelihood between drug exposure and the occurrence of ADR of health technologies in the post-marketing period.

Since the 1970s, different methods to standardize the evaluation of the causal association of ADRs have been available, ranging from small questionnaires to comprehensive algorithms[2].

The development of these tools, which are ordinary to use[3] and require minimal expertise to be employed[1,4], aims to solve methodological bias, reliability, and validity issues in the imputation of drug-induced adverse effects[5]. However, the main advantage arises from the possibility of decentralizing the causality assessment from the medical diagnosis, extending it to different health care levels: academics, the pharmaceutical industry, and health agencies[6].

By standardizing ADR causality assessment, the uncertainty of the association between a drug and an adverse event will not be reduced, but semi-quantitatively categorized[2] in different links of probability.

Establishing a causal link may influence the rationale for the correlation of an event that occurs to drug consumers[7]; therefore, the results of the causality assessments using algorithms must be reproducible. This is important to ratify the viability of their employment in pharmacovigilance[8], as well as their capacity to detect ADR signals[9,10]. This is because the higher the agreement on a defined ADR causal link, the more robust the hypothesis about the relationship between the use of a medication and the adverse event observed, allowing the communication of the risk and, therefore, the implementation of risk minimization and patient safety plans.

Because serious ADRs lead to hospitalization, it is necessary to assess the causality in the tertiary health care level. However, there are few data about the agreement on ADR causality assessment using different algorithms in patients hospitalized in internal medicine units in developing countries.

It is known that most ADR evidence arises from hospitals, due to the high risks associated with treatments in the tertiary health care level[11]. Therefore, the causality assessment in high complexity institutions contributes to: i) the early recognition of adverse effects, which helps to prevent iatrogenic complications; ii) therapy optimization[2]; iii) establishing barriers to prevent recurrence; iv) reducing the time of hospitalization and unnecessary burden with hospitalizations that could be avoided[12].

This study aimed to compare the results of the imputation of ADRs using different algorithms in a Brazilian public hospital, to identify the most appropriate for establishing causal associations between medication use and the occurrence of adverse events.

2 Material and methods

2.2 Selection of algorithms

Twenty-nine (29) algorithms for ADR causality assessment were identified by literature review. Nineteen (19) were excluded for the following reasons: absence of equivalent terminology for the level of imputation of ADRs (n = 6); inclusion of information that is not required for the causality assessment in Brazil (n = 3); tools that were developed for the assessment of specific ADRs (n = 3); and no access to the article (n = 7).

The ten algorithms (Table 1) considered eligible for the study included the combination of five main criteria for the causality assessment[13], namely: i) plausible temporality; ii) prior bibliographic description of the adverse effects related to the use of the drug involved; iii) alternative causes; iv) positive withdrawal (discontinuation of the drug with improvement of the ADR); v) positive rechallenge (reintroduction of the drug with reappearance of the ADR).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Algorithms selected for the causality assessment of adverse drug reactions in a public and general hospital in the State of Sao Paulo, Brazil (n = 10).

https://doi.org/10.1371/journal.pone.0171470.t001

Owing to the quantitative and qualitative variability in the terminologies used to express the results of the imputation of ADRs in the included algorithms, the nomenclature developed by Macedo et al. (2005)[13] was used. To improve the accuracy of the comparison, the equivalent terms of the likelihood level were grouped into four major categories: definite, probable, possible and unlikely.

2.4 Statistical analysis

Two descriptive statistics were used to measure the nominal agreement between two or more raters: Cohen´s kappa and Light´s kappa.

Cohen´s kappa measure the degree of concordance between two judges. The analysis carried out by FRV (rater A) was considered gold-standard to calculate the inter-rater agreement between judges B, C and D.

Light´s kappa is a multi-rater statistic which measures the degree of concordance among multiple judges without gold-standard. It is an extension of Cohen’s kappa. For both tests, we considered α = 0.05, 95%CI for all analyses. Values were interpreted according to Landis and Koch protocol (1977)[17] (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Interpretation of the Kappa (K) value.

https://doi.org/10.1371/journal.pone.0171470.t002

3 Results

During the period of data collection, the risk management department received 24 ADRs reports that enclose 36 different types of clinical manifestations resulting from 19 drugs (Table 3). Owing to the causality imputation was carried out case to case, each judge independently assess 44 cases, since a single report may describe more than one clinical manifestation associated with only one drug or may signalize more than one suspected drug for the occurrence of a single clinical manifestation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of the adverse drug reactions (ADR) analyzed, according to characteristic of patients, expectative, frequency, seriousness of the event (n = 44).

https://doi.org/10.1371/journal.pone.0171470.t003

According to seriousness, seven ADR reports showed symptomatology classified as serious, since 4 of them prolonged hospital length-stay, 2 resulted in temporary disability and 1 was related to hospital admission.

After causality assessment, none of the algorithms showed 100% agreement between judges on the imputation of ADRs. Fair agreement was observed for both statistic tests (Cohen´s and Light´s kappa) (Table 4). Findings suggest the poor reproducibility of the algorithms in performing ADR imputation with different judges.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Inter-rater and multi-rater agreement in adverse drug causality assessment, according to the statistical analysis with Cohen´s and Light´s kappa.

https://doi.org/10.1371/journal.pone.0171470.t004

The lower inter-rater agreement was observed for the judge B, except for Emanueli (1984), Jones (1986) and WHO-UMC (2010) algorithms (Table 4).

Venulet (k = 0.15), Kramer (k = 0.19), and Naranjo (k = 0.20) algorithms showed the worst multi-raters coefficient, indicating slight agreement on causal association. Moreover, the agreement was stronger in Emanueli (k = 0.36), Mashford (1984) and WHO-UMC (k = 0.36) algorithms, but not better than fair (Table 4).

4 Discussion

Our data suggest that WHO-UMC algorithm is the most consistent for causal imputation of hospital ADR that affected patients admitted to an internal medicine unit of a medium complexity hospital. The advantage of this tool is the semi-quantitative assessment of the causal likelihood and of the quality of the report; it has been used as a gold standard in causality studies[8,13]. Moreover, this tool was developed to evaluate the occurrence of adverse effects during the post-marketing period, which helps to achieve higher probability scores of causal association and a better reproducibility between the judges.

Emanueli (1984) algorithm, which contains a minimalist, simplified, dichotomous structure that considers only the clinical condition of the patient with alternative cause, may overestimate the cases of ADR and generate false-positive signals in risk communication, which is why it is not the most recommended for causality assessment in the context of this study.

For the remaining algorithms, we noted a weak agreement between the judges on ADR causality. Studies have shown a great variability in the results of imputation of ADR using different algorithms[8,13,18–22]. According to Shakir and Layton (2002)[10], the tools are inconsistent and sometimes of poor quality for signal detection. Furthermore, they have significant limitations that reduce the accuracy and reliability of the assessment of the probability of ADR[1].

Considering Naranjo et al. (1981) algorithm, data from previous study showed slight agreement between the judges[21], because it was developed and validated for the assessment of ADRs that occur during randomized clinical trials[19]. Other authors suggest the use of this tool for the imputation of ADR[21] due to its rapid implementation. However, we disagree this is the only factor to consider when choosing an algorithm. In addition to this aspect, the reliability of the results and the limitations of each tool, especially in the context of medication use (clinical trial versus post-marketing surveillance), should be considered. According to the data from our study, WHO-UMC (2010) partially meets these criteria.

We understand that it meets in an incomplete manner, because all of the analyzed algorithms do not include other factors that may be associated with adverse events, such as medication errors, product quality deviations, and suspected therapeutic ineffectiveness. Most consider in the assessment only the drug safety issues and neglect (Emanueli 1884; Blanc et al., 1979; Gallagher et al., 2011) or ambiguously (Karch Lasagna, 1977), subjectively (Naranjo et al., 1981; Mashford, 1984; WHO-UMC, 2010) or complexly (Kramer et al., 1979; Venulet et al., 1986) manage other factors that may be associated with adverse events. Even Gallagher et al. (2011) algorithm which was developed after the new definition of pharmacovigilance in 2002 did not include relevant information in the assessment, allowing underestimation of a causal association. This may also be correlated with the low agreement between the judges.

The arbitrary weighting given to the evaluation criteria is another limitation that may contribute to the inconsistency of algorithms[5,23]. This adds subjectivity inherent to the algorithm structure according to criteria these authors deem most important and give greater weighting in scoring. The causality assessment itself also includes some subjectivity[2,6]. Both situations described may contribute to the poor agreement between the algorithms in the imputation of the causal link of ADR.

Another evidence that may decrease the accuracy of risk communication is the absence of ADR reports of good quality and underreporting[10,24,25]. Poor or missing information in the reports makes it difficult assessing causality in details, differentiating between probable and possible cases[2], and finding a definitive causal association. Consequently, the assumptions are not robust enough to generate signals in pharmacovigilance, impairing the assessment of drug safety in the post-marketing period.

Nowadays, there is no gold-standard algorithm for the assessment of events occurring in primary care[6,26] and studies that compared the imputation of ADR reported to national pharmacovigilance centers[13,23]. At the tertiary health care level, Kane-Gill et al. (2012)[20] found strong agreement when comparing three algorithms by active search of retrospective cases of ADR in the intensive care unit. This can be explained by the ward where the study was conducted, the methodology (one judge) and the active search method. Critical patients are constantly monitored, so the records in medical charts are more complete, which allows the collection of better information and increases the robustness of causality assessment. However, the disadvantage of the active search is the time necessary to review medical records[27], which turns the process unfeasible.

Considering the limitations described, there is evidence that it is necessary to develop better quality tools that improve the diagnosis of ADR[26]. A strategy to increase the generation of pharmacovigilance signals is the monitoring of adverse drug events[28], and its evaluation criteria should be included in the algorithms to improve the reproducibility in the causal imputation. These criteria involve the assessment of any drug-related problem.

Therefore, in an attempt to minimize the described flaws and confounding variables during the assessment, the need, effectiveness, adherence and safety parameters should also be included in the algorithms in order to update the assessment with the new concepts of WHO (2002)[29] about post-marketing studies.

Considering drug use[28,30] and the intentional non-compliance with pharmacotherapy related to the diagnosis stereotypical diseases[31] are associated with undesirable effects, it is also necessary to evaluate the impact of ADRs on the patient. This would help to know the priorities, difficulties and factors that may motivate the use or discontinuation of therapy and therefore the occurrence of undesirable effects derived from these perceptions and practices.

Finally, although the literature presents a wide range of methods for the causality assessment, including computational approaches, algorithms are still viable alternatives to the causality assessment in hospitals, since these tools are easy to use, require little financial resources to be applied in the clinical routine and need minimal expertise to be applied[18]. Thus, it is important to update these algorithms in accordance with the new definition of pharmacovigilance, allowing the monitoring of adverse drug events[26], in order to minimize confounding variables associated with the causal imputation process and therefore improve the risk/benefit assessment of medications available on the market.

5 Conclusion

Our data show slight agreement on the ADR causality assessment for the majority of the tested algorithms. However, WHO-UMC (2010) algorithm showed fair reproducibility and allows the analysis of the quality of the report, which is why we suggest that it is the best tool for causality assessment of ADRs occurring in hospitals. Since the Naranjo algorithm was developed and validated to diagnose ADR occurring in randomized clinical trials and showed slight concordance between the judges, this tool is not the most consistent for the assessment of ADRs that affect non-critical patients in a secondary hospital. In addition, data demonstrate the need for the development of better quality tools, that include other criteria for the assessment of drug-related problems, such as effectiveness, safety, compliance, quality or quality deviation and medication errors.

Acknowledgments

The CAPES Foundation, Ministry of Education of Brazil for the scholarship (PDSE) grant n°. 014301/2013-00. The authors would also like to thank FAPESP for the financial support in this project, under the grant #2013/10263-9, São Paulo Research Foundation (FAPESP), the Programa de Apoio ao Desenvolvimento Científico da Faculdade de Ciências Farmacêuticas da UNESP-PADC and to Instituto de Biomedicina (iBiMED) FCT Ref N° UID/BIM/04501/2013. We are also thankful to the Hospital Estadual Américo Brasiliense, which allowed its data to be collected.