Skip to main content
  • Research article
  • Open access
  • Published:

Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study

Abstract

Background

Adverse drug events are a frequent cause of emergency department presentations. Administrative data could be used to identify patients presenting with adverse drug events for post-market surveillance, and to conduct research in patient safety and in drug safety and effectiveness. However, such data sources have not been evaluated for their completeness with regard to adverse drug event reporting. Our objective was to determine the proportion of adverse drug events to outpatient medications diagnosed at the point-of-care in emergency departments that were documented in administrative data.

Methods

We linked the records of patients enrolled in a prospective observational cohort study on adverse drug events conducted in two Canadian tertiary care emergency departments to their administrative data. We compared the number of adverse drug events diagnosed and recorded at the point-of-care in the prospective study with the number of adverse drug events recorded in the administrative data.

Results

Among 1574 emergency department visits, 221 were identified as adverse drug event-related in the prospective database. We found 15 adverse drug events documented in administrative records with ICD-10 codes clearly indicating an adverse drug event, indicating a sensitivity of 6.8% (95% CI 4.0–11.2%) of this code set. When the ICD-10 code categories were broadened to include codes indicating a very likely, likely or possible adverse event to a medication, 62 of 221 events were identifiable in administrative data, corresponding to a sensitivity of 28.1% (95% CI 22.3-34.6%).

Conclusions

Adverse drug events to outpatient medications were underreported in emergency department administrative data compared to the number of adverse drug events diagnosed and recorded at the point-of-care.

Peer Review reports

Background

Outpatient medication use is common, but may confer health risks that compromise its therapeutic benefits [13]. Health risks associated with medications have been shown to vary substantially in clinical practice from those observed in published randomized controlled trials [4, 5]. Capturing complete data on adverse drug events observed in clinical practice is important for post-market surveillance, drug regulatory activities, and research in drug safety and effectiveness and patient safety [68].

Emergency Departments play a pivotal role in North American healthcare systems [9]. They serve as acute diagnostic and treatment centers for ambulatory patients with unexpected and serious medical problems, as a safety net for the underserved and uninsured, and are an accessible portal of entry into acute care hospitals for sick patients. A growing proportion of urgent outpatient visits occur in emergency departments, and the majority of hospital admissions in the United States occur through emergency departments [9]. Ambulatory patients suffering from serious adverse drug events, the unexpected and unintended complications of medication use, commonly seek care in emergency departments [1015]. Therefore, emergency department administrative data may offer unique opportunities to track population-level data on clinically significant adverse drug events to outpatient medications for the purposes of surveillance and research [1620].

Administrative databases are readily available, inexpensive and can provide population-level health data on important health outcomes [8]. However, emergency department administrative data have not been evaluated for their completeness in adverse drug event reporting. Our objective was to determine the proportion of adverse drug events identified at the point-of care in two emergency departments by pharmacists and physicians that were recorded in administrative data.

Methods

Study design

We linked data from a prospective observational cohort study to emergency department administrative data [21]. We obtained data on patient demographics, emergency department visits, and adverse drug events to outpatient medications from the prospective study. We used each patient’s unique identifier and emergency department visit date to link the prospective study database with administrative databases to look for records of adverse drug events within the administrative data. The administrative databases used ICD-10 diagnostic codes, and one also recorded chief complaint codes. The University of British Columbia Clinical Research Ethics Board (H10-01632) reviewed and approved the study protocol, and waived the need for informed consent.

Setting

We used data collected as part of a prospective observational study that was conducted in two tertiary care emergency departments in Canada with a combined census of 145,000 patient-visits per year: Vancouver General Hospital (VGH) and St. Paul's Hospital (SPH).

Study cohort

Our study cohort consisted of patients enrolled into the prospective study. Patients were eligible for enrolment if they presented to the VGH or SPH emergency departments between July 1, 2008 and January 24, 2009. Research assistants enrolled patients using a previously described systematic patient selection algorithm to generate a representative sample. We included all English-speaking patients who were 19 years of age or older and reported using at least one prescription or over-the-counter medication within two weeks of presentation. In the prospective study, we excluded patients if they were agitated, presented with intentional self-poisoning, had previously been enrolled, presented for a scheduled revisit, had been transferred from another hospital, or left against medical advice or prior to being seen by the physician or pharmacist. When we linked the study databases we subsequently excluded the records of patients with multiple visits on the same day because they resulted in unresolved linkages.

Identification of adverse drug events at the point-of-care

One of three residency-trained clinical pharmacists who were research assistants in the prospective study and the treating emergency physician assessed each patient for adverse drug events in the emergency department in a manner that was independent and blinded to each other’s assessments using a pre-defined algorithm (Additional file 1): First, the pharmacists evaluated whether or not the patient’s visit was due to an adverse drug event using three adapted causality algorithms [2224]. The causality algorithms were used to standardize the pharmacists’ assessments, and to ensure that the exacerbation of underlying disease was considered as a cause for any events deemed potentially due to an adverse drug event. Inter-rater reliability of the pharmacists’ assessment using the causality algorithms was 0.75. After the pharmacy assessment was complete, we interviewed the treating emergency physician using a standardized questionnaire to determine the patient’s working diagnosis. When the physician and pharmacist determinations of a patient’s adverse drug event status were concordant (i.e., ADE/ADE or No ADE/No ADE), this was considered the criterion standard. If there was any disagreement between ratings (e.g., ADE/No ADE) or uncertainty (e.g., ADE/Uncertain), an independent committee consisting of a clinical pharmacist and a medical toxicologist (who was also a physician), both of whom were otherwise not involved in the study, adjudicated the cases (See Algorithm for adjudication committee, Additional file 1). If the case definition of adverse drug events was met, we categorized the events according to the Hepler & Strand taxonomy (Additional file 2) [25].

Identification of adverse drug events in the administrative data

We used administrative data that the Vancouver Coastal Health Authority routinely submits to the Canadian Institute for Health Information (CIHI) National Ambulatory Care Reporting System (NACRS). This database contains data on emergency department visits and includes the patient’s diagnosis at the time of discharge in addition to secondary diagnosis fields. We identified adverse drug events in the administrative data using a list of 650 ICD-10 codes generated through a review of the literature [26] and from Stausberg et al., [27, 28] and by searching for relevant chief complaint codes (Additional file 3). Our ICD-10 code set included codes that may indicate a manifestation of an adverse drug event (e.g., K25 Gastric ulcer) or an external cause (e.g., Y40-Y59 Drugs, medicaments and biological substances causing adverse effects in therapeutic use). To search for categories of adverse drug events not classified as adverse drug reactions, we also included Y66 (non-administration of surgical and medical care), T36-50 (poisoning by drugs, including overdose and wrong substance given or taken in error), T88.7-T88.9 (unspecified adverse events of a drug, and unspecified complications of medical care), Y57.9 (complications of medical and surgical care: drug or medicament, unspecified), Y69 (unspecified misadventure during surgical and medical care), and Y88.0 (sequelae of adverse effects caused by drugs in therapeutic use). The likelihood of an ICD-10 code representing an adverse drug event was based on the ICD-10 code description and on clinical reasoning (Table 1).

Table 1 Examples of ICD-10 codes indicating adverse drug events within each code category

Definitions

Adverse drug events were defined as “untoward and unintended symptoms, signs or abnormal laboratory values arising from the appropriate or inappropriate use of prescription or over-the counter medications” [3032]. All cases deemed adverse drug events had to be associated with an emergency department visit. Adverse drug events were distinguished from drug-related problems by the presence of untoward and unintended symptoms, signs or abnormal laboratory values. Once identified, adverse drug events were classified into mutually exclusive categories according to their etiology (Additional file 2) [25]. Those due to drug exposure were classified as: (1) adverse drug reactions, defined as noxious and/or unintended responses to medication which occurred despite appropriate drug dosage for prophylaxis, diagnosis or therapy of the indicating medical condition; [32] (2) drug interaction, (3) drug use without indication, or (4) supratherapeutic/high dose. Adverse events related to alcohol abuse and illicit drugs were not considered adverse drug events. Adverse drug events due to lack of exposure to a drug were classified as: (6) subtherapeutic/low dose, (7) need to add drug/untreated indication, (8) wrong drug, or (9) noncompliance/failure to receive drug [11, 30, 31, 33] Cases in which drugs were never considered for use in the first place were not considered adverse drug events. Adverse drug events were categorized into chief-complaint-related events versus those found incidentally. The former were deemed to result in the patient’s presenting complaint (i.e., a non steroidal anti-inflammatory drug leading to a gastrointestinal bleed, in which the patient’s complained of “vomiting blood”). The latter, were not deemed to result in the patient’s presenting complaint (i.e., hydrochlorothiazide leading to an abrupt and significant decline in serum sodium in a patient presenting with a skin infection). The emergency physician assigned an adverse drug event severity category at the point-of-care: A severe event caused death or required admission, a moderate event required a change in medical management, and a mild event required no change in therapy [2224]. The categorizations of adverse drug events by chief-complaint and severity were independent.

Analysis

We used descriptive statistics to summarize the baseline characteristics of the patient population. We estimated the proportion of patients with at least one adverse drug event code recorded in the administrative hospital or emergency department database among those who were diagnosed with one or more adverse drug events at the point-of-care. This was estimated as the number of patients with recorded adverse drug event codes in categories A1, A2, B1 or B2 in the administrative database, divided by the number of patients with adverse drug events diagnosed at the point-of-care in the prospective database multiplied by 100. We estimated the proportion of false positives by dividing the number of patients with at least one adverse drug event code recorded in the administrative database and no adverse drug events identified at the point-of-care by the number of all patients without adverse drug events identified at the point-of-care, multiplied by 100. As the likelihood of an ICD-10 code identifying a true adverse drug event varied, we expanded the numerator to include codes in categories C, D, or E.

Results

Adverse drug events identified at the point-of-care

Among 2289 patients who were approached for enrolment, 1591 met the prospective study’s inclusion and exclusion criteria (Figure 1) [21]. Among these, 17 were excluded because they had multiple visits on the same day in the administrative data leading to unresolved linkages.

Figure 1
figure 1

Patient flow.

The place of presentation was distributed unevenly between the two emergency departments: 1152 (73.2%) patients presented to VGH, and 422 (26.8%) to SPH. The average age was 51.4 years, 51.1% were female, and the median number of prescribed medications was two (Table 2). Among these patients, 221 (14%, 95% confidence interval 12.4–15.8%) were diagnosed with 237 adverse drug events at the point–of–care (Table 3). One hundred and forty six patients (146/221; 66.1% 95% CI 59.6–72.0%) had one adverse drug event related to their chief–complaint, 60 (60/221; 27.1 95% CI 21.7–33.4%) had one incidentally found event, and 15 (15/221; 6.8% 95% CI 4.2–10.9%) had more than one adverse drug event. The most common categories of adverse drug events resulted from adverse drug reactions (74/237; 33.5%; 95% CI 27.6–40.0%) (Table 3). Other categories of adverse drug events were due to noncompliance/failure to receive a drug (62/237; 26.2%; 95% CI 21.0-32.1%), need to add a drug/untreated indication (35/237; 14.8%; 95% CI 10.8-19.9%), subtherapeutic/low dose (27/237; 11.4%; 95% CI 8.0-16.1%), supratherapeutic/high dose (20/237; 8.4%; 5.5-12.7 95% CI), wrong drug (15/237; 6.3%; 95% CI 3.9-10.2%) and drug use without indication (4/237; 1.7%; 95% CI 0.7-4.2%). No adverse drug events were attributed to drug interactions.

Table 2 Characteristics of 1,574 patients presenting to the emergency department
Table 3 Characteristics of 237 adverse drug events identified at the point-of-care in 1574 emergency department patients

Most adverse drug events (226/237; 95.4%; 95% CI 91.9.0–97.4%) were deemed at least moderate in severity, requiring a change in medical management, consultation, hospital admission, or were life-threatening. The most commonly implicated medications were acetaminophen with codeine (15 events), warfarin (15 events) and phenytoin (9 events).

Proportion of adverse drug events reported in the administrative databases with codes clearly linking the event to a culprit medication

We found ICD-10 codes that clearly identified an adverse drug event (categories A1, A2, B1 and B2) in 15 of 221 records of patients diagnosed with one or more adverse drug events at the point–of–care (Table 4), corresponding to a sensitivity of 6.8% (95% CI 4.0–11.2%). This code set identified 18 of 1353 records as false positive (1.3%; 95% CI 0.8-2.1%), corresponding to a specificity of 98.7% (95% CI 97.9-99.2%). The positive predictive value of the code set was 45.5% (95% CI 28.1–63.7%), and its negative predictive value 86.6% (95% CI 84.8 – 88.3%).

Table 4 Patients with records in the administrative data indicating an adverse drug event among patients with adverse drug events identified at the point-of-care in the emergency department, by category of diagnostic ICD-10 code and type of adverse drug event

Two of 59 adverse drug reactions (3.4%; 95% CI 0.4–11.7%) and 3 of 22 severe adverse drug events (13.6%; 95% CI 2.9–34.9%) were identified as medication-related in the administrative data with code categories A1, A2, B1 or B2, among patients with only one event. Among patients admitted to hospital from the emergency department, 18.2% of adverse drug events were identified with an ICD-10 code clearly linking the event to medication use (Table 5). We found ICD-10 codes that clearly linked a culprit medication to an adverse drug event in 14 of 158 (8.9%, 95% CI 5.1–14.7%) records of patients presenting with a chief complaint-related adverse drug event.

Table 5 Patients with records in the administrative data indicating an adverse drug event among patients with adverse drug events identified at the point-of-care in the emergency department, by category of diagnostic code and admission status*

Adverse drug events identified in the administrative data with codes indicating a very likely, likely or possible relationship to a medication

When the ICD-10 code categories were broadened to include codes that very likely, likely or possibly indicated an adverse drug event, we were able to identify 62 codes for adverse drug events in the 221 records of patients diagnosed with one or more adverse drug events at the point–of–care (Table 4). This corresponded to a sensitivity of 28.1% (95% CI 22.3–34.6%) for the broader code set. The positive predictive value of the code set was 27.2% (95% CI 21.5 – 33.5%), and its negative predictive value 88.2% (95% CI 86.3 – 89.9%).

Using the broader code set we identified 23 of 59 adverse drug reactions (39.0%; 95% CI 26.6–52.6%), and seven of 22 severe adverse drug events (31.8%; 95% CI 13.9–54.9%) among patients with one event only. Among admitted patients, 54.6% of adverse drug events were identified with an ICD-10 code indicating a possible, likely, or very likely adverse drug event, or clearly linking the event to medication use (Table 5). The broader code categories identified 45 of 158 (28.5%, 21.7%–36.3%) adverse drug events related to the chief-complaint. This code set incorrectly identified 166 of 1353 records as false positive (12.3%; 95% CI 10.6-14.1%), corresponding to a specificity of 87.7% (95% CI 85.9-89.4%).

The most common culprit medications of all identified adverse drug events (categories A1–E) were olanzapine (five events), and warfarin, phenytoin, vancomycin, glyburide, clopidogrel and aspirin (two events each).

Discussion

Our objective was to determine the proportion of adverse drug events to outpatient medications resulting in emergency department visits that were reported in administrative data. We found adverse drug events to be underreported in the administrative data of two large Canadian university hospitals, despite using an extensive list of ICD-10 codes. Even when a broad set of ICD-10 codes was used to include diagnoses indicating a very likely, likely or possible relationship with a medication, we were only able to identify 28% of adverse drug events, and 39% of adverse drug reactions. Among patients who required hospital admission, we were able to identify 55% of adverse drug events.

Prescribing medication is the most common medical intervention performed by physicians. Yet, many prescribing decisions are informed by incomplete or conflicting evidence, or by the results of randomized trials that may not be transferrable to clinical practice [7, 34, 35]. In addition, off-label use of medications and varying compliance behavior of patients contribute to suboptimal treatment outcomes, leading to a growing interest in developing improved methods to capture adverse drug event data from the real-world to generate more robust estimates about the comparative safety and effectiveness of medications, and to develop interventions to improve patient care [68, 17, 18].

In North America, emergency departments offer the majority of healthcare delivered for acute and unexpected medical conditions, including adverse drug events [9]. Therefore, emergency department administrative data may offer unique opportunities to capture data on clinically significant adverse drug events that result from outpatient medication use [9, 1113]. However, before such data are considered for this purpose, they should be evaluated for their completeness.

Our study is the first in the peer-reviewed literature to compare adverse drug event reports in administrative records of emergency department patients with adverse drug events diagnosed at the point-of-care. Prior studies have attempted to validate adverse drug event codes in administrative data by comparing adverse drug event reports in administrative data to events identified by chart review, using electronic trigger methods or between administrative databases [3641]. Our study differs from these studies in the premise that all clinically significant adverse drug events are recorded in the medical record or identifiable using trigger methods. Indeed, two prior studies support the assumption that 40% of adverse drug events may not be documented in emergency department records [33, 42]. Therefore, in order to understand the sensitivity of administrative data and the ICD-10 code set, we derived our criterion standard at the point-of care using a pre-defined algorithm that included assessment by a pharmacist and a physician. We believe that this led to more precise estimates of adverse drug events. We disclosed all adverse drug events suspected in the emergency department to treating physicians (required by Ethics to ensure optimal patient care) prior to coding, thus optimizing the chances of their documentation in the medical chart.

A few studies have examined the sensitivity of adverse drug event codes within the ICD-9 coding system for events that occurred as a result of inpatient medications [38, 39]. Hougland et al. found that their code set detected more events than the hospital’s computerized adverse drug event surveillance system, and estimated that 55% of adverse drug events causing hospitalization, and 10% of adverse drug events occurring during the course of hospitalization were identified when compared to medical record review [38]. Leonard et al. found that the sensitivity of their ICD-9 code set varied substantially by the type of adverse drug reaction they searched for, and estimated that the sensitivity of the codes for digoxin and phenytoin related events may be 84% and 86.7% respectively [39]. However, the authors determined the criterion standard retrospectively by chart review in only 19-40% of records, all of which had been were pre-screened using an ICD-9 code set that included the adverse drug reaction codes [39]. This may have falsely elevated the sensitivity of their code set, because the determination of the criterion standard was not independent of the code set they used to identify events.

Only one previous study has evaluated the sensitivity of emergency department data coded in ICD-10 for adverse drug reactions [41]. Wu et al. used CIHI data to compare the emergency department discharge diagnosis with the admitting diagnosis among patients who were admitted to the hospital through emergency departments. The authors’ premise was that in patients admitted to hospital through the emergency department for a diagnosis of an adverse drug reaction, the patient’s emergency department discharge diagnosis and hospital admitting diagnosis should be the same, if adverse drug reactions are appropriately identified, recorded and coded. Using an ICD-10 code set containing 245 codes including the external cause codes Y40-59, Wu et al. found that 15% of emergency department visits for adverse drug reactions leading to hospital admission were coded with the corresponding admitting diagnosis in CIHI. In comparison, in our study including all emergency department patients (not just those admitted to hospital), we were only able to identify 3.4% of adverse drug reactions in the administrative data using our “narrower” code set containing ICD-10 codes categorized as A1, A2, B1 and B2. We believe that this large difference in our estimate of the degree of underreporting may be due to Wu et al.’s comparison of adverse drug reactions coded within one set of administrative data (NACRS) to another (the Discharge Abstract Database), as opposed to our comparison with a prospective standard. This indicates that adverse drug event reporting may be overestimated when reporting is evaluated by comparing between two administrative databases.

The strengths of our study include a rigorous assessment of adverse drug events at the point-of-care before any administrative coding occurred. Both a clinical pharmacist and a treating emergency physician assessed all patients in our cohort. The clinical pharmacists in our study evaluated patients independently from physicians, and took their own medical histories, contributing to the accuracy of the available medication information rather than relying on retrospective chart review. Pharmacists documented any suspected adverse drug events in the patients’ records and informed physicians of all potentially missed cases. All cases in which the pharmacists’ and physicians’ assessments of adverse drug events were discordant or uncertain were reviewed and adjudicated by an independent committee consisting of a clinical pharmacist and a medical toxicologist. Another strength of our study includes having conducted a literature review to identify adverse drug event codes in the ICD-10 coding system, reducing the possibility that we underestimated the capacity of the ICD-10 coding system to identify adverse drug events by using too narrow of a code set [26].

The operational definition of adverse drug events remains problematic, as several interpretations of its most common definition “harm caused by the use of a drug” exist [30, 31]. We approached our case definition of adverse drug events from the health services research perspective, in which the utilization of the emergency department leading to bed occupancy and incurring cost was the primary end point. Thus, all our cases were associated with an emergency department visit, and we did not capture any “harm” or injury” from illnesses not associated with an emergency department visit. Despite this, not all of the events captured in our study will be of interest from a pharmacovigilance or regulatory body perspective. From the latter perspective, adverse drug reactions, a subset of adverse drug events, are most relevant. Examples of events falling into our case definition that may not be relevant from a pharmacovigilance or regulatory body perspective were the following: the need to add a drug/untreated indication (e.g., lack of anticoagulation therapy leading to stroke in a patient with a previously established diagnosis of atrial fibrillation, a high CHADS2 score and previous documentation of the need for anticoagulation), too high or too low dose (e.g., a reduction in furosemide dose leading to pulmonary edema in a patient with previously controlled congestive heart failure and no alternate explanation), noncompliance/failure to receive a drug (e.g., noncompliance with insulin leading to diabetic ketoacidosis) or wrong drug (e.g., a patient with type II diabetes mellitus with recurrent episodes of hypoglycemia on glyburide). We deemed the inclusion of these types of events important from a health services research perspective, as these types of events have previously been associated with increased health services utilization and cost, [43] and many were classified as preventable [21]. From a patient, clinician and system perspective, the development of methods to identify and monitor these types of events is desirable to generate a factual basis for generating hypotheses about their prevention and to inform health policies to reduce their occurrence. These may include specific actions related to prescribing, administering or monitoring of high-risk medications, or actions targeting specific patient groups. Data on the extent of occurrence and associated burden of events can be used to prioritize actions in a resource-constrained environment to target commonly occurring preventable and costly events. For example, through a recently implemented adverse drug event screening program in the Vancouver Costal Health Authority, through which detailed regional adverse drug event data are collected, our group identified that a large proportion of emergency department visits can be attributed to supratherapeutic/high warfarin dose without any associated bleeding. Identifying the etiologic cause of these visits is informing the development of specific preventative policies within the Health Authority, as well as the discourse between primary care and acute care in terms of the etiology of outpatient adverse drug events and measures for prevention.

In order to ensure that we did not apply too broad of a case definition of adverse drug events, we put mechanisms in place to ensure that events that could be explained by the exacerbation of the patient’s underlying disease or by alternate diagnoses were excluded. These mechanisms included capturing the physician’s working diagnosis, mandating the use of causality algorithms and using an independent adjudication committee. We did not consider the failure to use drug in the first place as an adverse drug event, unless the drug had clearly been documented as being indicated in the patient’s medical record. We put these safeguards in place, as adopting too broad of a case definition and overcalling cases as adverse drug events that might not be, may risk promoting inappropriate use of non-medicinal therapies, and may not serve to promote prudent and rational medication use.

Our study is not without limitations. First, we considered all adverse drug events that had been reported in either primary or secondary diagnostic codes, because some patients presented to the emergency department with more than one adverse drug event, one of which may have been coded under a secondary diagnosis field. Also, some patients may have been diagnosed with more than one diagnosis in the emergency department, one of which was deemed the primary reason for presentation or admission. Therefore, we did not exclude adverse drug event codes that were coded in secondary diagnosis fields in order to avoid underestimating the sensitivity of the administrative dataset and the ICD-10 code set. However, this means that we may have picked up adverse drug events that resulted from in-hospital treatment rather than from outpatient medications. This would have resulted in an overestimation of the sensitivity of the ICD-10 codes. Second, because of the cost and labor involved in establishing a prospective standard for adverse drug events, our sample size is limited. Thus, our study should be regarded as preliminary. Third, our results reflect two Canadian institutions and may not be generalizable to other institutions. Fourth, we expanded our code set to include possible adverse drug events (code categories C, D and E), to allow for better ICD-10 data capture which resulted in a greater proportion of false positives. Fifth, our results may have been influenced by the existing variation in the use of the terminology surrounding adverse drug events [31]. It is possible that physicians were less likely to record (and coders less likely to code) events that they personally felt should not be considered drug-related, even though the presentation met our outcome definition. Finally, we wish to clarify why the number of events listed in this study differs from its parent study [21]: The prospective data used for this study was derived from a prospective observational clinical decision rule derivation study in which we collected data on all outcomes (n = 221). The purpose of the parent study was to derive clinical decision rules to aid health care workers at the point-of-care to identify patients with a broad range of adverse drug events. Yet, this was not possible, likely due to the heterogeneity of the events. Therefore, as stated a priori in the protocol of our parent study, we proceeded to derive clinical decision rules for two narrower categorizations of adverse drug events. Thus, the clinical decision rule derivation study represents a subset of the events analyzed in the present study.

Conclusion

We found adverse events to outpatient medications resulting in emergency department visits to be underreported in existing administrative data of two large Canadian tertiary care hospitals. The performance characteristics of the code sets examined, in terms of their sensitivity, specificity, positive and negative predictive values, indicate that administrative data alone may not be appropriate as a stand-alone means of identifying adverse drug events in these data. This study may serve as a point of reference for future work in this area, considering the paucity of the literature on evaluating the ICD-10 system and emergency department administrative data for adverse drug event reports. Future research on differential reporting of outpatient adverse drug events in administrative data would be useful to gain a better understanding of those events for which administrative data may have a greater sensitivity [38]. Finally, despite the cost and labor involved in establishing adverse drug event surveillance systems using prospective data, active case finding methods may result in more complete and accurate estimates compared with administrative data.

References

  1. Bates D: Drugs and adverse drug reactions. How worried should We Be?. JAMA. 1998, 279 (15): 1216-1217. 10.1001/jama.279.15.1216.

    Article  CAS  PubMed  Google Scholar 

  2. Patterns of medication use in the United States 2004: a report from the slone survey. http://www.bu.edu/slone/files/2012/11/SloneSurveyReport2006.pdf.

  3. Gurwitz JH, Field TS, Harrold LR, Rothschild J, Debellis K, Seger AC, Cadoret C, Fish LS, Garber L, Kelleher M, et al: Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003, 289 (9): 1107-1016. 10.1001/jama.289.9.1107.

    Article  PubMed  Google Scholar 

  4. Sorensen SV, Dewilde S, Singer DE, Goldhaber SZ, Monz BU, Plumb JM: Cost-effectiveness of warfarin: trial versus “real-world” stroke prevention in atrial fibrillation. Am Heart J. 2009, 157: 1064-1073. 10.1016/j.ahj.2009.03.022.

    Article  PubMed  Google Scholar 

  5. T-Pv S, Leufkens HG, Zhang B, Smeeth L: A comparison of cost effectiveness using data from randomized trials or actual clinical practice: selective Cox-2 inhibitors as an example. PLoS Med. 2009, 6 (12): e1000194-10.1371/journal.pmed.1000194.

    Article  Google Scholar 

  6. Wiktorowicz M, Lexchin J, Paterson M, Mintzes B, Metge C, Light D, Morgan S, Holbrook A, Tamblyn R, Zaki E, et al: Research networks involved in post-market pharmacosurveillance in the United States, United Kingdom, France, New Zealand, Australia, Norway and European Union: Lessons for Canada. Edited by: the Canadian Institute for Patient Safety. 2008, Edmonton, Canada: The Canadian Institute for Patient Safety.

  7. Developing a Protocol for Observational Comparative Effectiveness Research (OCER): A User’s Guide (XXX under Contract No. XXX HHSA.). http://www.effectivehealthcare.ahrq.gov/ehc/products/440/1067/AHRQ_CER_Protocol_User_s_Guide_DRAFT-COPY_AllChapters.pdf.

  8. Suissa S, Garbe E: Primer: administrative health databases in observational studies of drug effects—advantages and disadvantages. Nat Clin Pract Rheum. 2007, 3 (12): 725-732. 10.1038/ncprheum0652.

    Article  CAS  Google Scholar 

  9. Schuur JD, Venkatesh AK: The growing role of emergency departments in hospital admissions. N Engl J Med. 2012, 367: 391-393. 10.1056/NEJMp1204431.

    Article  CAS  PubMed  Google Scholar 

  10. Hafner JW, Belknap SM, Squillante MD, Bucheit KA: Adverse drug events in emergency department patients. Ann Emerg Med. 2002, 39 (3): 258-267. 10.1067/mem.2002.121401.

    Article  PubMed  Google Scholar 

  11. Hohl CM, Dankoff J, Colacone A, Afilalo M: Polypharmacy, adverse drug-related events, and potential adverse drug interactions in elderly patients presenting to an emergency department. Ann Emerg Med. 2001, 38 (6): 666-671. 10.1067/mem.2001.119456.

    Article  CAS  PubMed  Google Scholar 

  12. Zed PJ, Abu-Laban RB, Balen RM, Loewen PS, Hohl CM, Brubacher JR, Wilbur K, Wiens MO, Samoy LJ, Lacaria K, Pursell RA: Incidence, severity and preventability of medication-related visits to the emergency department: a prospective study. Can Med Ass J. 2008, 178 (12): 1563-1569. 10.1503/cmaj.071594.

    Article  Google Scholar 

  13. Budnitz DS, Lovegrove MC, Shehab N, Richards CL: Emergency Hospitalizations for Adverse Drug Events in Older Americans. N Engl J Med. 2011, 365: 2002-2012. 10.1056/NEJMsa1103053.

    Article  CAS  PubMed  Google Scholar 

  14. WHO: Nternational drug monitoring: the role of national centres. Report of a WHO meeting. 1972, Geneva: World Health Organization, 1-44.

    Google Scholar 

  15. Zed PJ: Drug-related visits to the emergency department. J Pharm Pract. 2005, 18: 329-335. 10.1177/0897190005280049.

    Article  Google Scholar 

  16. Budnitz DS, Pollock DA, Weidenbach KN, Mendelsohn AB, Schroeder TJ, Annest JL: National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006, 296 (15): 1858-1866. 10.1001/jama.296.15.1858.

    Article  CAS  PubMed  Google Scholar 

  17. Carleton B, Foerster V, Warren L, Smith A: Overview of novel drug plan and drug regulatory pharmacosurveillance initiatives in the United States, United Kingdom, and select other jurisdictions: A background paper prepared for the Working Conference on Strengthening the Evaluation of Real World Drug Safety and Effectiveness. September 13-15, 2005

    Google Scholar 

  18. Keeping an eye onprescription drugs, keeping Canadians safe: active monitoring systems for drug safety and effectiveness in Canada and internationally. http://publications.gc.ca/collections/collection_2011/ccs-hcc/H174-21-2010-eng.pdf.

  19. Budnitz DS, Pollock DA, Mendelsohn AB, Weidenbach KN, McDonald AK, Annest JL: Emergency department visits for outpatient adverse drug events: demonstration for a national surveillance system. Ann Emerg Med. 2005, 45 (2): 197-206. 10.1016/j.annemergmed.2004.09.020.

    Article  PubMed  Google Scholar 

  20. Jhung MA, Budnitz DS, Mendelsohn AB, Weidenbach KN, Nelson TD, Pollock DA: Evaluation and Overview of the National Electronic Injury Surveillance System-Cooperative Adverse Drug Event Surveillance Project (NEISS-CADES). Med Care. 2007, 45 (10): S96-S102.

    Article  PubMed  Google Scholar 

  21. Hohl CM, Brubacher J, Hunte G, Wiens MO, Abu-Laban R, Singer J, Sheps S: Clinical Decision Rules to Improve the Detection of Adverse Drug Events in Emergency Department Patients. Acad Emerg Med. 2012, 19 (6): 640-649. 10.1111/j.1553-2712.2012.01379.x.

    Article  PubMed  Google Scholar 

  22. Forster A, Asmis T, Clark H, Al Saied G, Code C, Caughey S, Baker K, Watters J, Worthington J, van Walraven C: Ottawa Hospital Patient Safety Study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. Can Med Ass J. 2004, 170 (8): 1235-1240. 10.1503/cmaj.1030683.

    Article  Google Scholar 

  23. Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, Janecek E, Domecq C, Greenblatt DJ: A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981, 30 (2): 239-245. 10.1038/clpt.1981.154.

    Article  CAS  PubMed  Google Scholar 

  24. World Health Organization: Letter MIO/372/2(A). 1991, Geneva, Switzerland: WHO

    Google Scholar 

  25. Hepler C, Strand L: Opportunities and responsibilities in pharmaceutical care. Am J Hosp Pharm. 1990, 47: 533-543.

    CAS  PubMed  Google Scholar 

  26. Karpov A, Stausberg J, Doyle-Waters M, Hohl CM: Adverse drug event codes in ICD-10: a systematic review of the literature. In press

  27. Stausberg J, Hasford J: Identification of adverse drug events: the use of ICD-10 coded diagnoses in routine hospital data. DtschArztebl Int. 2010, 107 (3): 23-29.

    Google Scholar 

  28. Stausberg J, Hasford J: Drug-related admissions and hospital-acquired adverse drug events in Germany: a longitudinal analysis from 2003 to 2007 of ICD-10-coded routine data. BMC Health Serv Res. 2011, 11: 134-10.1186/1472-6963-11-134.

    Article  PubMed  PubMed Central  Google Scholar 

  29. International statistical classification of diseases and related health problems 10th revision. http://apps.who.int/classifications/icd10/browse/2010/en.

  30. Edwards IR, Aronson JK: Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000, 356 (9237): 1255-1259. 10.1016/S0140-6736(00)02799-9.

    Article  CAS  PubMed  Google Scholar 

  31. Nebeker J, Barach P, Samore M: Clarifying adverse drug events: a clinician’s guide to terminology, documentation, and reporting. Ann Intern Med. 2004, 140: 795-801. 10.7326/0003-4819-140-10-200405180-00017.

    Article  PubMed  Google Scholar 

  32. WHO: International drug monitoring: the role of the hospital. Report of a WHO meeting. WHO Technical Report Series. 1966, 425: 5-24.

    Google Scholar 

  33. Hohl CM, Robitaille C, Lord V, Dankoff J, Colacone A, Pham L, Berard A, Pepin J, Afilalo M: Emergency physician recognition of adverse drug-related events in elder patients presenting to an emergency department. Acad Emerg Med. 2005, 12 (3): 197-205. 10.1111/j.1553-2712.2005.tb00869.x.

    Article  PubMed  Google Scholar 

  34. Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA: Practical clinical trials for translating research to practice design and measurement recommendations. Med Care. 2005, 43: 551-557. 10.1097/01.mlr.0000163645.41407.09.

    Article  PubMed  Google Scholar 

  35. Tunis SR, Stryer DB, Clancy CM: Practical clinical trials. Increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003, 290 (12): 1624-1632.

    Article  CAS  PubMed  Google Scholar 

  36. Verelst S, Jacques J, Heede KV, Gillet P, Kolh P, Vleugels A, Sermeus W: Validation of hospital administrative dataset for adverse event screening. Qual Saf Health Care. 2010, 19: e25.

    PubMed  Google Scholar 

  37. Hougland P, Nebeker J, Pickard S, Tuinen MV, Masheter C, Elder S, Williams S, Xu W: Using ICD-9-CM codes in hospital claims data to detect adverse events in patient safety surveillance. 2008, Agency for Healthcare Research and Quality: Rockville (MD), Volume 1.

    Google Scholar 

  38. Hougland P, Xu W, Pickard S, Masheter C, Williams SD: Performance of international classification of diseases, 9th revision, clinical modification codes as an adverse drug event surveillance system. Med Care. 2006, 44: 629-636. 10.1097/01.mlr.0000215859.06051.77.

    Article  PubMed  Google Scholar 

  39. Leonard CE, Haynes K, Localio R, Hennessy S, Tjia J, Cohen A, Kimmel SE, Feldman HI, Metlay JP: Diagnostic E-codes for commonly used, narrow therapeutic index medications poorly predict adverse drug events. J Clin Epidemiolog. 2008, 61: 561-571. 10.1016/j.jclinepi.2007.08.003.

    Article  Google Scholar 

  40. Drösler S, Romano P, Wei L: Health Care Quality Indicators Project: Patient Safety Indicators Report 2009. Organization for Economic Co-operation and Development (OECD) Health Working Papers No 47. Edited by: Head of Publications Service, OECD. 2009, Paris, France: Head of Publications Service, OECD, 1-47.

  41. Wu C: Adverse Drug Reactions in the Emergency Department Population in Ontario: Analysis of National Ambulatory Care Reporting System and Discharge Abstract Database 2003–2007. 2009, Toronto: University of Toronto

    Google Scholar 

  42. Hohl CM, Zed PJ, Brubacher JR, Abu-Laban RB, Loewen PS, Purssell R: Do emergency physicians attribute drug-related emergency department visits to medication-related problems?. Ann Emerg Med. 2010, 55 (6): 493-502. 10.1016/j.annemergmed.2009.10.008.

    Article  PubMed  Google Scholar 

  43. Hohl CM, Nosyk B, Zed P, Kuramoto L, Sobolev B, Brubacher J, Abu-Laban R, Loewen PS, Sheps S: Outcomes of emergency department patients presenting with adverse drug events. Ann Emerg Med. 2011, 58 (3): 270-279. 10.1016/j.annemergmed.2011.01.003.

    Article  PubMed  Google Scholar 

Pre-publication history

Download references

Role of the funding source

This study was supported by grants from the Canadian Patient Safety Institute, the Michael Smith Foundation for Health Research and the Vancouver Coastal Health Authority. None of the sponsors had any role in study design, data collection or processing, analysis or preparation of the manuscript. All authors had access to the study data and agreed to submit the manuscript in its current form. Dr Hohl is supported by a New Investigator Award from the Canadian Institutes of Health Research.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Corinne M Hohl.

Additional information

Competing interests

None of the authors have identified competing financial or non-financial interests.

Authors’ contributions

CH, BS and JS have made substantial contributions to the conception and design of the study, CH and EY aquired the data, LK and BR analyzed the data, and all authors interpreted the results of data analysis. CH, LK, BS and JS were involved in drafting and critically reviewing the manuscript. All authors have approved the manuscript in its current form.

Electronic supplementary material

12913_2012_2907_MOESM1_ESM.docx

Additional file 1: ADE Evaluation Algorithm at the Point-of-Care. In the ED each patient was evaluated by a clinical pharmacist and the treating emergency physician independently, and blinded to each other’s evaluations. The ratings were combined while the patient was still in the ED. If there was any disagreement about the rating (i.e., yes/no, yes/uncertain, no/uncertain, etc.), or if either or both of the evaluations were uncertain, the case proceeded to independent adjudication by a committee consisting of a pharmacist and physician not in any other way involved in the study. (DOCX 55 KB)

12913_2012_2907_MOESM2_ESM.docx

Additional file 2: Adverse drug events were diagnosed only in patients presenting to the emergency department with untoward and unintended symptoms, signs or abnormal laboratory values that arose from appropriate or inappropriate medication use. Events meeting this case definition were categorized according to the taxonomy of drug-related problems [25]. For all events that were categorized as due to “Need to Add Drug/Untreated Indication” or “Failure to Receive a Drug/Noncompliance” a pre-existing diagnosis had to have been documented prior to the emergency department visit. Failure to use a drug in the first place was not considered an adverse drug event. Asymptomatic drug-related problems were not captured. Adverse drug reactions were defined according to the World Health Organization [31]. (DOCX 51 KB)

Additional file 3: See code set provided in an Excel spreadsheet.(XLSX 34 KB)

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Hohl, C.M., Kuramoto, L., Yu, E. et al. Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study. BMC Health Serv Res 13, 473 (2013). https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6963-13-473

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6963-13-473

Keywords