Figures
Abstract
Introduction
Ethiopia achieved a rapid expansion of TB microscopic centers for acid fast bacilli (AFB). However, external quality assurance (EQA) services were, until recently, limited to few regional and sub-regional laboratories. In this paper, we describe the decentralization experience and the result of EQA using random blinded rechecking.
Materials and Methods
The routine EQA quarterly report was compiled and analyzed. A positive result by the microscopic center while the EQA center reported negative result is categorized as false positive (FP). A negative result by the microscopic center while the EQA center reported positive is considered false negative (FN). The reading of EQA centers was considered a gold standard to compute the sensitivity, specificity, positive predictive (PPV) and negative predictive values (NPV) of the readings of microscopic centers.
Results
We decentralized sputum smear AFB EQA from 4 Regional Laboratories (RRLs) to 82 EQA centers and enrolled 956 health facilities in EQA schemes. Enrollment of HFs in EQA was gradual because it required training and mentoring laboratory professionals, institutionalizing internal QA measures, equipping all HFs to perform diagnosis, and establishing more EQA centers. From 2012 to 2014 (Phase I), the FP rate declined from 0.6% to 0.2% and FN fell from as high as 7.6% to 1.6% in supported health facilities (HFs). In HFs that joined in Phase II, FN rates ranged from 5.6 to 7.3%. The proportion of HFs without errors has increased from 77.9% to 90.5% in Phase I HFs and from 82.9% to 86.9% in Phase II HFs. Overall sensitivity and specificity were 95.0% and 99.7%, respectively. PPV and NPV were 93.3% and 99.7%, respectively.
Citation: Melese M, Jerene D, Alem G, Seid J, Belachew F, Kassie Y, et al. (2016) Decentralization of Acid Fast Bacilli(AFB) External Quality Assurance Using Blind Rechecking for Sputum Smear Microscopy in Ethiopia. PLoS ONE 11(3): e0151366. https://doi.org/10.1371/journal.pone.0151366
Editor: Mark Spigelman, Hebrew University, ISRAEL
Received: October 14, 2015; Accepted: February 26, 2016; Published: March 18, 2016
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: All relevant data are within the paper.
Funding: United States Agency for International Development(USAID) supported this work through HEAL TB, under cooperative agreement number AID-663-A-11-00011. The contents of the article are the responsibility of the authors alone and do not necessarily reflect the views of USAID or the US government.
Competing interests: The authors have declared that no competing interest exist.
Introduction
Direct sputum microscopy for acid-fast bacilli (AFB) using light microscopy is the most widely used tuberculosis (TB) diagnostic and monitoring tool worldwide [1–4]. Quality-assured TB microscopy is one of the key elements of DOTS in the STOP TB strategy of the World Health Organization (WHO) [4]. It is simple and cost effective and does not require sophisticated training or setup [4–6]. But it does require a very good system of quality assurance [6–7].
Quality assurance consists of quality control (QC), external quality assurance (EQA), and quality improvement (QI). To yield reliable, reproducible results, all three components should be implemented across the laboratory network [8]. Reliable AFB microscopic results such as smear positivity rates also help planners to understand the progress of TB control measures [9–11]. Implementation of EQA for microscopy helps to improve the quality of diagnosis of TB and measure the cure rates of TB patients on treatment. EQA is needed to ensure that smears are performed and interpreted correctly and that all microscopy centers perform at an acceptable level [5,11,12].
Despite rapid expansion of TB microscopic centers in Ethiopia for Ziehl-Neelsen (ZN), EQA services were, until recently, limited to a few regional and sub-regional laboratories. To fill this gap, the Ethiopian Public Health Institute (EPHI) introduced a decentralized EQA system using randomized blinded rechecking (RBRC) [7]. RBRC involves the collection of smears from the microscopy center laboratory for blinded re-reading at a regional reference laboratory (RRL) or other designated EQA center, with feedback to the microscopy center [7]. WHO has recommended this approach to evaluate the performance of AFB microscopy centers [13].
RBRC has been used successfully in many pilot and research projects [9,13–16]. In India, for example, RBRC has been used to measure the performance of laboratories and assess errors [13,16]. In other settings, it has been used for QI of diagnosis and monitoring of treatment response [17,18] and for QA where culture or fluorescent microscopies cannot be routinely used [19]. In Ethiopia, we supported the implementation of a decentralized EQA system for ZN microscopy over 1,600 health facilities (HFs) in two large regions. This paper presents the process of decentralization, its outcomes, and the factors that contributed to successful establishment of RBRC services.
Materials and Methods
Setting
In Ethiopia, which is one of the 22 high-TB-burden countries,5 the Federal Ministry of Health (FMOH) provides guidance for implementation of the national TB program, while the EPHI is responsible for all laboratory-related standardization and quality issues. In 2008, EPHI designed an EQA system for sputum smear Z-NAFB microscopy [7]. The system was organized so that EPHI conducts panel testing for RRLs and the RRLs conduct RBRC of sputum smear slides for hospitals. Selected hospitals with good EQA performance (≥ 95% concordance for 2–3 quarters) conduct EQA for health centers in their catchment areas (Fig 1).
Operationalization of decentralized EQA
The Amhara and Oromia Regional Health Bureaus (RHBs) receive support from the US Agency for International Development (USAID) through the Help Ethiopia Address the Low Performance of TB (HEAL TB) project managed by Management Sciences for Health. The RHBs operationalized the decentralized EQA model through a process that involved several stakeholders.
When the project began in 2011, QA measures for AFB were weak and in most cases HFs had no QA mechanism. Following a baseline assessment, HEAL TB supported the RHBs to design a decentralized EQA system. The support included training laboratory personnel, providing standard registers, supplying microscopes and reagents, and providing quarterly supportive supervision and on-site technical support to every HF. During supervision visits, laboratory experts checked for complete registration and proper storage of the sputum smear slides and sequential labeling. The senior laboratory expert assisted in establishing internal QA measures during the site visits, including weekly checking of reagent quality with five known negative and positive slides. When errors were identified, the laboratory experts explored the cause of the error and took corrective measures with the HF laboratory personnel. After two or three quarters of follow-up and on-site support, the HFs were ready to join the country’s RBRC scheme. To assure their quality, reagents were prepared at national or regional level and distributed to all health facilities.
Training of district TB focal persons as supervisors and slide randomization
Once the HFs were prepared for EQA participation, trained woreda (district) TB focal persons took the lead in supervising them. The woreda TB focal persons were trained in supervisory skills for laboratories, including checking for proper registration, labeling, and storage of slides. Every quarter the woreda TB focal persons supervised each HF in their catchment area and randomized slides for blinded rechecking following the Lot Quality Assurance Sampling guidelines for AFB slides. The nationally agreed-on sample size is based on 80% sensitivity, 100% specificity, and accepting number d = 0 (Table 1) [20]. The TB focal persons then delivered the collected slides to the RRLs or EQA hospitals to conduct the EQA. The EQA readers were laboratory experts from the RRLs or hospitals, who were different from those who randomized the slides.
Blind re-checking procedures
The experts involved in EQA reading have demonstrated a 95% concordance rate for at least two quarters. EPHI assesses the RRLs’ EQA through panel testing, since EPHI does not routinely collect slides for patient care. RRLs check hospitals’ EQA quarterly, and designated hospitals check the EQA of health centers (Fig 1). The quality officer of the EQA center assigns slides to a reader (controller). After the first reader completes the reading, the result is submitted to the quality officer to reconcile with the initial reading of the microscopic center. The quality officer assigns all discrepant slides from the first reader to a second reader (senior expert). If the result of the second reader agrees with that of the first, it becomes the final result. If the readings are still discordant, a third expert reads the slide, and any two concordant expert readings become the EQA result. The RRLs or hospital controllers travel to health facilities with discordant slides to identify the cause of the discordance (e.g., poorly functioning microscope, reader capacity, quality of reagent, or fading). The final result is recorded after agreement with the HF laboratory professionals is reached. (Fig 2). If the final result is different from the original report of the microscopic center, it is communicated to the treating clinicians for decision making Each EQA center covers 11–15 HFs (Table 2), and controllers conduct EQA mostly in their spare time and are paid overtime. EQA reading takes one month, and on-site evaluations of HFs with discordant slides, takes another two months.
Data management and analysis
The reading results of peripheral laboratories were entered in Excel, the EQA results tabulated, and the analysis done based on internationally accepted definitions. The false-positive and false-negative errors were calculated using standard definitions [21]. A positive result by the microscopic center while the EQA center reported negative result is categorized false positive (FP). Similarly, if the microscopic center indicates a negative result while the EQA center reported positive, it is considered false negative (FN). Sensitivity, specificity, and positive and negative predictive values of the readings were then calculated using the EQA center controller’s final result as a gold standard, per the international guideline [20].
Ethical Considerations
The Ethiopian Public Health Institute (EPHI) has released an AFM microscopy EQA guideline to be implemented in all microscopy centers and using a Lots Quality Assurance, sputum smear slides collected through to the routine clinical practice are randomized for RBRC. The data for this paper is acquired through this routine lab quality monitoring system, but not collected from patients directly for research purpose. As per the guiding, the sputum smear slides randomized have no patient identification information and the result is reported to evaluate the lab performance, but not directly related to patient management. EPHI has given the permission to publish the experience from the nationally reported data as the practice of decentralized EQA system has much application for low-income countries.
Results
Baseline data
During the first phase of project implementation (July 2011-June 2013), 691 DOTS-providing HFs were supported through HEAL TB in Amhara and Oromia regions. At baseline in October 2011, 465 HFs were providing TB diagnostic services, and 104 were participating in sporadic AFB on-site quality checks but not RBRC using proper sampling. By the end of 2013, the remaining 226 non-diagnostic HFs were equipped to become diagnostic and all 629 were enrolled in RBRC. During the second phase of the project (July 2013-present), 909 more HFs were included and at the time of this analyses 335 of them were enrolled in RBRC. The rest were under mentorship to be part of the RBRC scheme.
Trends in EQA participation
The number of diagnostic HFs participating in RBRC increased from none at baseline to 956 by the end of March 2014. Ninety-one percent of the 691 DOTS centers in Phase I HFs were able to participate in the quarterly RBRC scheme, while the remaining 9% were checked on site because of low slide volume. Of the Phase II HFs, 37% have started to participate in EQA. In June 2014, the number of EQA-participating facilities decreased because the HFs with heavy patient loads shifted from Ziehl-Neelsen microscopy to iLED (light-emitting diode) fluorescence microscopy. Enrollment of HFs in EQA was gradual because it required training and mentoring laboratory professionals, institutionalizing internal QA measures, equipping all HFs to perform diagnosis, and establishing more EQA centers (Table 2).
The EQA centers grew from 4 at baseline to 82 by the end of June 2014 (Table 2). Between April 2012 and June 2014, the EQA centers had read 248,832 slides. In Phase I HFs the false-positive rate declined from 0.6% (95% CI, 0.4–0.7) to 0.2% (95% CI, 0.2–0.3) and false negatives had a steady decline from 7.6% (95% CI, 6.1–9.6) to 1.6% (95% CI, 1.0–2.6) over two years, with a slight increase in the last quarter in Phase I HFs (Table 3, Fig 3). The denominator used to calculate the false negative is positive readings and for that false positives is negative readings. The proportion of HFs with no errors at this increase in Phase I reached 90.5% as opposed to 77.9% at the beginning of the project (Fig 4). In Phase II HFs the false-negative rate ranged from 5.6% to 7.3% while false positives ranged from 0.5% to 0.3% (Table 3, Fig 3).
By the end of the study, overall sensitivity and specificity for the Phase I HFs were 95% and 99.7%, respectively, and the positive predictive value (PPV) and negative predictive value (NPV) were 93.7% and 99.7% respectively. In Phase II HFs, sensitivity and specificity were 94.1% and 99.6% respectively. The PPV and NPV were 93.3% and 99.7% respectively.
In Phase I HFs, the average quality of staining at baseline was 71.1% and by June 30, 2014, it reached 81.4%. In Phase II HFs, it increased from 61.7% at baseline to 72.7% by June 2014. Smear thickness also improved, from 62.1% to 69.8% in Phase I HFs, but in Phase II HFs it improved from 59.3% to 71% and then decreased to 57.0%. In Phase I HFs cleanliness of the slides improved from 72.6% to 86.3%, but in Phase II HFs cleanliness improved from 80.7% to 88.4% and then declined to 81.6% in June 2014 (Table 4).
Discussions
This study demonstrates that decentralizing AFB EQA services to hospitals is a feasible, low-cost approach for countries like Ethiopia that have few higher-level laboratories [8,13,22,23]. With the decentralized approach, nearly 1,000 HFs participated in RBRC. It takes approximately 6–9 months to prepare the HFs for RBRC, but in Phase II, 335 HFs enrolled in EQA, which was faster than expected because of the experience gained during Phase I implementation.
The rapid capacity building of HFs in sputum smear microscopy, coupled with on-site supervision, helped decrease the numbers of false-positive and false-negative slides. Other countries have reported improvements using similar mechanisms [8,18] Our experience is that the proportion of HFs with no errors improved from quarter to quarter but the error rate fluctuated because of HFs enrolled in EQA for the first time or new, less-trained laboratory professionals assigned to the HFs. The HFs began EQA in different phases, but in three quarters the false-negative rate declined significantly (Table 3). In 507 Phase I HFs with EQA results, for example, in April-June 2014, only 48 contributed to the reported errors. False-positive errors were low from the beginning, and there was statistically significant improvement in both phases.
Another possible reason for errors was smear quality, although there were improvements from quarter to quarter because of comprehensive capacity building (Table 4). We addressed challenges by providing refresher training at sites with poor performance and at laboratories with new personnel. In addition, the EQA centers served as mentors and trainers for new laboratory professionals and underperforming laboratories. Every week the HF also checks reagent quality with known negative and positive slides. RRL staff also visit and identify the causes of errors with the HFs’ laboratory experts. If fading is suspected, they re-stain the slides and read them on-site with the same microscope used for diagnosis by the HF.
The overall sensitivity of 95.0% and specificity of 99.7% in our health facilities are high, per international standards [20], and the national recommendation about the sample size for RBRC for Ethiopia might need revision. The revised international recommendation for EQA of AFB smear microscopy is to use a 75–80% sensitivity rate to calculate sample sizes for blinded rechecking [7,20]. Ethiopia has already adopted this recommendation, so the sample size for blinded rechecking was calculated based on a sensitivity of 80% [7]. Future samples will be large, if Ethiopia plans to revise the sampling based on the improved EQA results. EQA centers may be overloaded with large numbers of slides.
The decentralized approach is cost effective because the EQA readers and district TB focal persons who randomize slides for EQA are all government workers paid according to government rates. EQA readers are paid US$5 per 20 slides in Oromia Region and US$2 in Amhara. The per diem and transport for the district focal person is US$7.50 per HF. These costs are manageable for the government, which will help to sustain the system. The experience in scaling-up of EQA, the progressive improvement in quality and the cost-effectiveness of the approach heralds that such system can easily be easily replicated in similar settings.
There are some limitations of the study. The false-positive and false-negative rates at EQA center level were not reported using the scanty, 1+, 2+, and 3+ categories, but for four quarters in the initial period the regions were reporting summary data, so we could not compute error rates by category. Therefore the analysis is limited to false positives and false negatives rather than detailed classifications. The data did not capture whether the AFB slides included in the EQA were collected for diagnostic purpose or TB treatment follow-up. As a result, we were not able to compare the EQA in the two groups independently. However, the regularity of data collection and the huge number of HFs covered represent strengths of this study.
Conclusion
A decentralized EQA scheme was feasible in a large number of HFs in Ethiopia. Involving hospitals has contributed to rapid scale-up of the EQA scheme to thousands of HFs every quarter. AFB quality has improved gradually and error rates have declined in many HFs. The model is scalable and sustainable because it was designed and built within the Ethiopian health care system. Close on-site mentoring of DOTS centers and of HFs with errors are critical for the success of this approach. Pre-placement trainings for newly assigned laboratory personnel should be implemented routinely to prevent the high error rates reported from sites with new staff. Smear quality improvement is a priority to further reduce errors. The overall impact of the decentralized EQA scheme on improving the quality of TB care should be evaluated. A clear sputum sample transport to the expanding GeneXpert and culture centers should be established to improve the diagnosis of TB those cannot be by microscopy.
Acknowledgments
We thank the FMOH, Amhara and Oromia RHBs, RRLs, health workers, laboratory professionals, and TB program coordinators for their active role in implementing this project. Alaine Umubyeyi Nyaruhirira and Sara Holtz reviewed the final draft of the paper, and Barbara K. Timmons edited the manuscript.
Author Contributions
Conceived and designed the experiments: MM JS YK BG DH DJ NH. Analyzed the data: MM DJ DH G. Ayana G. Alem JS FB SN YK YKH NH PGS. Contributed reagents/materials/analysis tools: MM DJ DH G. Ayana G. Alem JS FB SN YK YKH BG PGS. Wrote the paper: MM DJ DH G. Ayana JS FB SN YK YKH PGS.
References
- 1. Ridderhof J C, van Deun A, Kam K M, Narayanan P R, Aziz M A. Roles of laboratories and laboratory systems in effective tuberculosis programmes. Bull World Health Organ 2007; 85: 354–359. pmid:17639219
- 2. American Thoracic Society. Diagnostic standards and classification of tuberculosis in adults and children. Am J Respir Crit Care Med 2000; 161: 1376–1395. pmid:10764337
- 3.
World Health Organization (WHO). Tuberculosis handbook. WHO/TB/98253. Geneva, Switzerland: WHO; 1998.
- 4.
World Health Organization (WHO). The Stop TB Strategy: Building on and enhancing DOTS to meet the TB-related Millennium Development Goals. WHO/HTM/TB/2006368. Geneva, Switzerland: WHO, 2006.
- 5. Tuberculosis Division, International Union Against Tuberculosis and Lung Disease. Tuberculosis bacteriology—priorities and indications in high prevalence countries: position of the technical staff of the Tuberculosis Division of the International Union Against Tuberculosis and Lung Disease. Int J Tuberc Lung Dis. 2005; 9: 355–361. pmid:15830740
- 6. Van Denu A, Portaels F. Limitations and requirements for quality control of sputum smear microscopy for acid-fast bacilli. Int J Tuber Lung Dis. 1998; 2: 756–765.
- 7.
Ethiopian Health Nutrition and Research Insititute. Guideline for quality assurance of smear microscopy for TB diagnosis. Addis Ababa, Ethiopia: Federal Ministry of Health; 2009.
- 8. Van Rie A, Fitzgerald D, Kabuya G, Van Deun A, Tabala M, Jarret N, et al. Sputum smear microscopy: evaluation of impact of training, microscope distribution, and use of external quality assessment guidelines for resource-poor settings. J Clin Microbiol. 2008; 46: 897–901. pmid:18174302
- 9. Patel ND, Rade K, Dave PV, Pujara K, Solanki RN, Vegad MM, et al. Impact of the RNTCPIRL-EQA-OSE visits on quality of sputum smear microscopy services of Gujarat, India. Indian J Tuberc. 2012; 59: 12–17. pmid:22670506
- 10. Gilpin C, Kim S J, Lumb R, Rieder H L, Van Deun A, Working Group on Sputum Smear Microscopy. Critical appraisal of current recommendations and practices for tuberculosis sputum smear microscopy. Int J Tuberc Lung Dis. 2007; 11: 946–952. pmid:17705970
- 11. Mundy C J, Harries A D, Banerjee A, Salaniponi F M, Gilks C F, Squire S B. Quality assessment of sputum transportation, smear preparation and AFB microscopy in a rural district in Malawi. Int J Tuberc Lung Dis. 2002; 6: 47–54. pmid:11931401
- 12. Kusznierz G F, Latini O A, Sequeira M D. Quality assessment of smear microscopy for acid-fast bacilli in the Argentine tuberculosis laboratory network, 1983–2001. Int J Tuberc Lung Dis. 2004; 8: 1234–1241. pmid:15527156
- 13. Otero L, Van Deun A, Agapito J, Ugaz R, Prellwitz G, Gotuzzo E, et al. Quality assessment of smear microscopy by stratified lot sampling of treatment follow-up slides. Int J Tuberc Lung Dis. 2011; 15: 211–216. pmid:21219683
- 14. Chiang CY, Rieder HL, Kim SJ, Kam KM, Dawson D, Lin TP, et al. Quality of sputum smear microscopy in Taiwan. J Formos Med Assoc. 2005; 104: 502–506. pmid:16091827
- 15. Manyazewal T, Paterniti A D, Redfield R R, Marinucci F. Role of secondary level laboratories in strengthening quality at primary level health facilities’ laboratories: an innovative approach to ensure accurate HIV, tuberculosis, and malaria test results in resource-limited settings. Diagn Microbiol Infect Dis. 2013; 75: 55–59. pmid:23102548
- 16. Selvakumar N, Prabhakaran E, Rahman F, Chandu NA, Srinivasan S, et al. Blinded rechecking of sputum smear for acid-fast bacilli to ensure the quality and usefulness of restaining smears to assess false positive errors. Int J Tuberc Lung Dis 2003; 7: 1077–1082. pmid:14598968
- 17. Martinez A, Balandrano S, Parissi A, Zuniga A, Sanchez M, Ridderhof J, et al. Evaluation of new external quality assessment guidelines involving random blinded rechecking of acid-fast bacilli smears in a pilot project setting in Mexico. Int J Tuberc Lung Dis. 2005; 9: 301–305. pmid:15786894
- 18. Wu M H, Chiang C, Jou R, Chang S Y, Luh K T. External quality assessment of sputum smear microscopy in Taiwan. Int J Tuberc Lung Dis. 2009; 13: 606–612. pmid:19383194
- 19. Yip C W, Chan M Y, Cheung W F, Yu K W, Tang H S, Kam K M. Random blind rechecking of sputum acid-fast bacilli smear using fluorescence microscopy: 8 years’ experience. Int J Tuberc Lung Dis. 2012; 16: 398–401. pmid:22640454
- 20.
World Health Organization, Centers for Disease Control and Prevention, Association of Public Health Laboratories (APHL), KNCV and International Union Against Tuberculosis and Lung Disease. External quality assessment for AFB smear microscopy. Washington, DC: APHL; 2002.
- 21.
Jekel J F, Katz D L, Elmore J G, Wild D. Epidemiology, biostatistics and preventive medicine. 3rd ed. Philadelphia, PA: Elsevier Health Sciences, 2007.
- 22. Kumar V, Raghavan R, Nagamiah S, Chauhan L S. External quality assessment of smear microscopy by the National Reference Laboratory in nine states of India. Int J Tuberc Lung Dis. 2009; 13: 1183–1185. pmid:19723411
- 23. Martinez-Guarneros A, Balandrano-Campos S, Solano-Ceh MA, Gonzalez-Dominguez F, Lipman HB, Ridderhof JC, et al. Implementation of proficiency testing in conjunction with a rechecking system for external quality assurance in tuberculosis laboratories in Mexico. Int J Tuberc Lung Dis. 2003; 7: 516–521. pmid:12797692