Figures
Abstract
This study developed and validated the Youth Conduct Problems Scale-Rwanda (YCPS-R). Qualitative free listing (n = 74) and key informant interviews (n = 47) identified local conduct problems, which were compared to existing standardized conduct problem scales and used to develop the YCPS-R. The YCPS-R was cognitive tested by 12 youth and caregiver participants, and assessed for test-retest and inter-rater reliability in a sample of 64 youth. Finally, a purposive sample of 389 youth and their caregivers were enrolled in a validity study. Validity was assessed by comparing YCPS-R scores to conduct disorder, which was diagnosed with the Mini International Neuropsychiatric Interview for Children, and functional impairment scores on the World Health Organization Disability Assessment Schedule Child Version. ROC analyses assessed the YCPS-R's ability to discriminate between youth with and without conduct disorder. Qualitative data identified a local presentation of youth conduct problems that did not match previously standardized measures. Therefore, the YCPS-R was developed solely from local conduct problems. Cognitive testing indicated that the YCPS-R was understandable and required little modification. The YCPS-R demonstrated good reliability, construct, criterion, and discriminant validity, and fair classification accuracy. The YCPS-R is a locally-derived measure of Rwandan youth conduct problems that demonstrated good psychometric properties and could be used for further research.
Citation: Ng LC, Kanyanganzi F, Munyanah M, Mushashi C, Betancourt TS (2014) Developing and Validating the Youth Conduct Problems Scale-Rwanda: A Mixed Methods Approach. PLoS ONE 9(6): e100549. https://doi.org/10.1371/journal.pone.0100549
Editor: Tom Denson, The University of New South Wales, Australia
Received: November 15, 2013; Accepted: May 28, 2014; Published: June 20, 2014
Copyright: © 2014 Ng et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This research was supported by the Peter C. Alderman Foundation, the Harvard Center for the Developing Child, and the Julie Henry Faculty Development Fund. This publication was supported by grants 1K01MH07724601 A2, 1R34MH084679-01 A1 and R34MH084679-03S1 from the National Institute of Mental Health, the Harvard Center for the Developing Child, the Julie Henry Development Fund, and the FXB Center for Health and Human Rights. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The prevalence of youth mental health disorders in sub-Saharan Africa (SSA) is estimated at approximately 10% [1]. These disorders pose a high public health burden due to their association with social and functional impairments [2] and the “youth bulge” in SSA [3]. Prevalence of conduct problems have been found to be fairly consistent across diverse cultures and contexts [4]. Among mental health disorders, conduct problems are thought to affect 3.1% to 5.7% of youth, with a median age of onset between 7 and 15 [5]. Generally, across cultures, research has found that boys have more conduct problems than girls [6], [7].
Accurate and valid tools for assessing youth conduct problems are critical, since they are associated with a host of poor outcomes including academic failure [8], violence, antisocial behavior [9], injuries, substance abuse [10], [11], [12], sexual risk behaviors [13], [14], [15], sexual and physical victimization [16], and psychiatric comorbidities, including mood and anxiety disorders [17], [18]. However, little is known about the differences in symptom presentation and perception of youth conduct problems in low and middle income countries [3]. To our knowledge, no tools for assessing conduct problems have been developed for SSA youth. This study developed and validated the Youth Conduct Problems Scale-Rwanda (YCPS-R).
The YCPS-R was developed to evaluate the efficacy of an intervention for youth with an HIV positive caregiver. Research in many countries indicates that children with HIV-positive caregivers are at increased risk for a range of mental health problems including conduct problems due to disrupted parent-child relationships, fear, and misinformation [19], [20], [21], family conflict, stigma, economic insecurity, lower education achievement, and caregiver depression and physical impairment [22], [23], [24]. Although the overall HIV prevalence rate in Rwanda has been declining and is currently estimated to be 3% [25], this trend masks the high rates of HIV among adults of caregiving age (7.9% of females between 35–39), and that one-in-six children in Rwanda is classified as vulnerable due to HIV and AIDS [26], [27]. Given the large number of children in Rwanda impacted by HIV, and their subsequent increased risk for conduct problems, developing context-appropriate measures of conduct problems, is critical for identifying children and assisting them with accessing services and care.
Method
The YCPS-R was developed using a mixed-methods approach [28], [29] to ensure that items were culturally and contextually relevant. Qualitative studies identified and compared local symptoms and presentation profiles of youth conduct problems to scales validated in higher resource settings. Quantitative studies assessed the reliability and validity of the YCPS-R.
Ethics Statement
Study protocols were approved by the Rwanda National Ethics Committee and the Harvard School of Public Health's Institutional Review Board. All adults gave written informed consent for themselves, all caregivers gave written informed consent for their children, and all children gave written informed assent for themselves.
Phase 1: Identification of Mental Health Problems
To identify local expressions of conduct problems, the research team conducted two qualitative studies in Southern Kayonza district following methods previously published [30]. The first was a series of free list (FL) interviews with 31 adults (42% female) and 43 youth (47% female) selected to capture a range of gender, age, and HIV serostatus. The second gathered key informant (KI) interviews on problem themes from 47 participants. FL participants identified and described as many problems facing HIV/AIDS-affected youth in their community as possible. Problem themes representing mental health and psychosocial issues were noted. KIs, identified by FL participants and community members as being knowledgeable about psychosocial issues facing HIV/AIDS-affected youth, elaborated on problem themes and identified common cover terms that described distinct local mental health syndromes which represented specific constellations of symptoms. Ten Clinician KIs who were Rwandan mental health professionals, pediatricians, and social workers (60% women) reviewed the syndromes and associated indicators to assist with defining syndromes from a clinical perspective. One major goal of the Clinician KIs was to refine distinctions between syndromes to ensure the most accurate categorization of symptoms.
Phase 2: Measure Selection and Development
The qualitative process described in phase one resulted in the identification of a local conduct problems syndrome and its associated symptoms. These culturally-specific symptoms were compared to items on existing measures of conduct problems to determine whether the local symptoms overlapped with indicators from measures validated in other contexts. An extensive search of youth conduct problem measures was conducted using Ovid's Health and Psychosocial Instrument database, PubMed, Google Scholar, and reference lists from identified publications. Search criteria included prior cross-cultural application with youth, strong psychometric properties, and evidence of validity in low-resource, low-literacy settings. Items from standard measures were compared for conceptual equivalence to local conduct problem indicators by U.S. and Rwandan research teams. Scales were considered a promising match if at least 50% of local indicators were captured. If a match was not made, the YCPS-R was developed solely from local indicators.
Phase 3: Cognitive Testing
The YCPS-R was cognitive tested in Kirehe district (which borders Southern Kayonza) by six youth (aged 11–17, three females) selected from villages and schools by village leaders and principals, and six caregivers (four females). The goal of cognitive testing was to understand how participants interpreted items and chose responses. Following cognitive testing guidelines [31], bilingual (Kinyarwanda and English) Rwandan research assistants (RAs), all of whom had a bachelor's degree, assessed participant comfort with item content and asked follow-up questions to examine comprehension (“What do you think this question is asking?”), retrieval (“Tell me what you were thinking when you gave your answer?” “How did you come up with your answer?”), and judgment (“Why did you answer that way?”). RAs transcribed and translated participant answers verbatim. The study team reviewed the data and discussed items requiring revision.
Phase 4: Reliability Study
Reliability of the YCPS-R was tested using a sample of 64 youth (51.56% female) aged 10–17 (mean age = 12.72, SD = 2.03) who were enrolled in school and living in Kirehe district. Participants were selected by village leaders and school principals. Exclusion criteria were cognitive impairment that interfered with item comprehension (as assessed by study psychologists) and refusal of caregiver consent or youth assent. The YCPS-R was administered orally by one of three Rwandan RAs and Cronbach's alpha was run to assess internal consistency. To assess whether participants would provide consistent responses to the same RA and to different RAs, one to three days later, half of the youth were administered the scale again by the same RA to assess test-retest reliability, and the remaining youth were administered the scale by a different RA to assess inter-rater reliability. Test-retest and inter-rater reliability were measured by Intraclass Correlation Coefficient (ICC) scores and Spearman's rho correlations using SPSS version 18 software [32]. Detailed demographic information has been published previously [28].
Phase 5: Validity Study
Validity of the YCPS-R was assessed by 389 youth aged 10–17 (44.15% female; mean age = 13.41, SD = 2.24) from Southern Kayonza district and one of their caregivers. Youth were excluded if they had lived in the region for less than a month, did not speak Kinyarwanda, or had cognitive impairment as described above. In order to measure the scale's ability to differentiate between youth with and without conduct problems, the enrollment goal was to recruit at least 50 participants who likely had each syndrome and at least 50 non-cases. To identify children with and without disorders, Community Advisory Board (CAB) members and community health workers (CHWs were asked to identify youth in their villages thought to have at least one of the local mental health syndromes identified during phase one (including conduct problems), and youth thought to have no mental health disorders. In Rwanda, every family is assigned to a CHW who oversees the health and wellbeing of all of the family members. Each CHW is responsible for approximately 50 families and all CHWs were able to identify children in their villages who did or did not have at least one of the local mental health syndromes. Of the 389 youth, 53 (13.62%) were identified by either a CAB member or a CHW as having the local conduct problems syndrome, and 130 (33.42%) were reported by CAB members or CHWs to be free of any mental health syndromes.
The validity study protocol has been described previously [28]. Briefly, four bachelor's level Rwandan RAs administered the YCPS-R to youth and caregiver participants, and participants also reported (via a yes/no response) whether they thought the youth had the local conduct problems syndrome. Locally validated versions of the Center for Epidemiological Studies Depression Scale for Children (CES-DC) [28] and the World Health Organization Disability Assessment Schedule Child Version (WHODAS Child) [29] a measure of functional impairment, were also administered. One to three days later, two Rwandan bachelors-level psychologists administered the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI KID) [33] including the conduct disorder (CD) and oppositional defiant disorder (ODD) subscales, to all validity study youth participants, which acted as the study's diagnostic “gold standard.” Small changes were made to the MINI KID items to make them more relevant to youth in rural Rwanda. For example, for objects that youth might use as weapons, “knives” and “guns” were retained from the original MINI KID, but “bats,” an object not typically found in Rwanda, was changed to “sticks” and “machetes.” Interviews were conducted verbally, privately and individually in participants' homes, and interviewers were blinded to the possible syndrome status of the youth. After completion of assessments, each family was given a small household gift worth approximately US$2.
Internal reliability of the YCPS-R was measured with Cronbach's alpha. Construct validity was assessed through factor analysis, with a hypothesized factor structure of one common construct. Criterion validity was assessed using correlations between the YCPS-R, MINI KID CD and ODD diagnoses, and reported presence of the local conduct syndrome, as well as between the YCPS-R and functional impairment. Discriminant validity was assessed through correlations with depression, a construct that is comorbid with conduct disorder [18], [34]. ROC curve analyses determined sensitivity and specificity trade-offs between the YCPS-R and the CD and ODD diagnoses. The Youden Index, which maximizes the difference between the true positive and false positive rate, was used to determine the optimal threshold point on the ROC curve. Differences in ROC results by sex and age were analyzed. Validity study analyses were conducted using STATA version 12 software [35].
Phase 6: Short Form Development
To create a short form of the scale, youth responses from the validity study were analyzed for removal. Items were removed if: 1) over 90% of participants endorsed the highest or lowest response option; 2) inter-item correlations were high (the item that better represented the construct per local clinician feedback was kept, and the other was dropped); or 3) if bootstrapped factor loadings were low. The short-form was assessed for internal reliability, criterion and discriminant validity, and classification accuracy.
Results
Phase 1: Identification of Mental Health Problems
Analysis of the FL and KI interview data identified “uburara,” or delinquent behavior, as the local term for a cluster of youth conduct problems, including being unruly, roaming around, and engaging in sexual intercourse. Twenty-one indicators of uburara were identified (see Table 1). Some uburara indicators approximated DSM-IV-TR [36] symptoms of CD and ODD based on a face validity review by Rwandan psychologists and the U.S.-based research team. However only seven of the 21 uburara indicators corresponded to symptoms of CD or ODD, and only four of eleven CD symptoms and three of seven ODD symptoms were captured by uburara indicators. These results suggest that uburara is a related but distinct manifestation of youth conduct problems in Rwanda (see Table 1).
Measure Selection and Development
Seven standardized measures of youth conduct problems were identified from the literature search. However, none had items that overlapped at least 50% with the uburara indicators. Thus, the YCPS-R was developed solely from the uburara indicators. Fifteen indicators were selected as items for the YCPS-R per feedback from Rwandan psychologists, and “Roaming around” and “Wandering” were separated into two items, resulting in a 16-item scale. Six items were dropped because they were not seen as hallmark criteria of uburara or were thought to be largely due to comorbidity with other conditions or social context: “Disappearing from home (runs away)/Not wanting to stay at home,” “Lacking a good conscience,” “Lacking good parenting,” “Not being grateful for what is given to him/her,” “Grumbling/Keeping a grudge,” and “Feeling everyone around is mocking him/her.”
A corresponding YCPS-R caregiver report on the youth was also developed (e.g. My child engaged in bad behaviors). The YCPS-R asked youth and their caregivers to respond to each item based on how the youth felt and acted during the past week. Response options were scored on a four point Likert-type scale (0 = Not at all, 1 = A little, 2 = Some, and 3 = A lot), and the scale score was the sum (possible range 0–48). The one week time frame was selected to correspond with the time frame used in the Center for Epidemiological Studies Depression Scale for Children (CES-DC) [37], which was being validated in the same study [28]. The YCPS-R required little forward translation since local Kinyarwanda indicators were used as items. Back translation was conducted according to best-practice protocols [38], [39].
Phase 3: Cognitive Testing
All YCPS-R items were understood by participants during cognitive testing, but six items (“I play dangerously/I am delinquent,” “I can't stay at home/I roam around,” “I use/take drugs,” “I am fearless,” “I dropped out of school although I have means/money,” “I am not clean though I have the means”) were edited to improve comprehension. For example, “I play dangerously” was removed from “I play dangerously/I am delinquent,” since participants only responded to the delinquency aspect, and “I am not clean though I have the means” was changed to specify having “hygiene materials” rather than having “means.” See Table 2 for the final YCPS-R.
Phase 4: Reliability Study
Results of the reliability study indicated that YCPS-R scores were skewed, with 28.13% of youth reporting no conduct problems (mean = 5.61, SD = 7.11; range = 0 to 41). Internal consistency of the reliability study data was good, with a Cronbach's alpha of .88. Of the 64 youth, 34 were enrolled in the test-retest reliability study, and the remaining 30 were enrolled in the inter-rater reliability study. The test-retest ICC was .56 and the rs was .54, p<.001. The ICC of the inter-rater reliability sample was .88, and the rs was .68, p<.001.
Phase 5: Validity Study
Results of the validity study found that 23.55% of caregivers and 10.11% of youth reported that the youth had uburara, 19.95% of youth were diagnosed with CD (27.54% of males, 10.39% of females), and 3.66% were diagnosed with ODD (1.52% of males and 5.88% of females). Results from the validity study indicated that the Cronbach's alpha was .90 for the youth report and .94 for the caregiver report. YCPS-R scores were skewed, with a youth-reported mean (SD) of 8.87 (9.34), median of 6, and a range of 0 to 42.
Construct, Criterion, and Discriminant Validity.
The YCPS-R had one common factor, accounting for 87.02% of the variance in the youth report and 88.42% in the caregiver report. YCPS-R scores were associated with CD diagnoses, and community, youth, and caregiver reported uburara diagnoses, but not with ODD diagnoses. YCPS-R scores were also highly correlated with functional impairment. The correlation between youth-reported YCPS-R and functional impairment was .58, and between caregiver reported YCPS-R was .63, indicating that the higher children scored on the YCPS-R, the poorer their functioning, and the greater their disability. YCPS-R scores had good discriminant validity, with significant correlations between conduct problems and depression ranging from .24 to .55 (see Table 3 for all correlations).
Sex and Age.
Results of logistic regressions found that youth reported (t(364) = 4.71, p = .03) and caregiver reported (t(360) = 9.98, p = .002) YCPS-R scores were higher in males than in females. Similarly, males were more likely to be diagnosed with CD (B = −1.19, p<.001), to have community referred uburara (B = 1.61, p<.001), caregiver reported uburara (B = −.95, p = .001), and to have self-reported uburara (B = −.98, p = .02) than females. However, females were more likely to be diagnosed with ODD than males (B = 1.40, p = .04). Results of logistic regressions indicated that age did not predict diagnosis of CD or community, caregiver, or youth reported uburara diagnoses (ps>.24), and results of linear regression found that age did not predict youth or caregiver reported YCPS-R scores (ps>.12).
Classification Accuracy.
ROC analyses assessed the ability of the YCPS-R to discriminate between youth diagnosed and not diagnosed with CD. Since only 3.66% of youth were diagnosed with ODD, ROC analyses were restricted to CD diagnoses. Since sex predicted YCPS-R and CD, ROC analyses were run by sex. Age was not included as a covariate, since age did not predict YCPS-R or CD. There were no differences in the youth report YCPS-R predicting CD under the curve (AUC) for males and females (X2 = .61, p = .44). The AUC was .75 (95% CI = .68–.82), which is considered reasonably accurate [40] (see Figure 1). The Youden Index optimal threshold was 13, which provided a sensitivity of 57.14% and a specificity of 84.70%. The AUC for the caregiver report YCPS-R was higher for females (AUC = .78, 95% CI = .65–.90) than males (AUC = .62, 95% CI = .54–.71) (X2 = 3.95, p<.05) (see Figure 2). The optimal threshold for the caregiver report was nine for females (sensitivity = 75.00%, specificity = 72.59%) and 16 (sensitivity = 54.39%, specificity = 65.75%) for males.
Phase 6: Short Form of the YCPS-R
Analysis of validity study item distributions indicated that more than 90% of responses to the items “I engaged in fornication/prostitution” and “I took/drank drugs” were 0 = Not at all, and so they were dropped from the short-form scale. The items “I roamed around” and “I wandered” were conceptually similar and highly correlated (rs = .72, p<.001). The local assessment team thought “I roamed around” better represented the uburara construct, and so it was kept, while “I wandered” was dropped. The items “I was not clean even though I didn't lack hygiene materials” and “I felt hopeless” loaded least well onto the construct (λ = .52 and .62, respectively), and so these items were dropped. See Table 2 for the complete short-form scale.
Cronbach's alpha for the short form was .89 for the youth report and .93 for the caregiver report. The short form also demonstrated good criterion validity, as the youth and caregiver reports were correlated with youth (rs = .23, rs = .23 respectively) and caregiver (rs = .28, rs = .62) reported functional impairment, CD diagnoses (rs = .35, rs = .28) and community (rs = .18, rs = .28), caregiver (rs = .28, rs = .67), and youth reported uburara diagnoses (rs = .34, rs = .29) (all ps<.005). The youth and caregiver report short form also demonstrated good discriminant validity, as they were moderately correlated with youth (rs = .46, rs = .28) and caregiver reported youth depression (rs = .21, rs = .51, all ps<.001).
ROC analyses found no difference between the AUC for males and females for the youth report short form (X2 = .47, p = .49). Classification accuracy of the youth report short form was fair (AUC = .75, 95% CI = .69–.82), and the optimal threshold was five (sensitivity = 77.14%, specificity = 63.35%). The AUC for the caregiver reported short form was better for females (AUC = .79, 95% CI = .68–.91) than males (AUC = .63, 95% CI = .54–.71) (X2 = 4.90, p = .03). The optimal threshold for the caregiver reported short form was nine for females (sensitivity = 68.75%, specificity = 80.00%) and 14 for males (sensitivity = 49.12%, specificity = 71.92%).
Discussion
The psychometric properties of the YCPS-R were strong, with one common factor and good internal, test-retest, and inter-rater reliability. The measure also demonstrated good criterion validity, as scores were positively associated with functional impairment, and self-reported, caregiver-reported, and community-referred uburara diagnoses, as well as “gold standard” CD diagnoses from the MINI KID. The YCPS-R also demonstrated good discriminant validity by being correlated, but not collinear with, depression.
The classification accuracy of the youth report YCPS-R and the caregiver report for females was fair, but the classification accuracy of the caregiver report for males was poor. The fair to poor classification accuracy may reflect differences between uburara and the MINI KIND CD diagnosis that was used as the gold standard. Indeed, the CD diagnosis may not be appropriate in the Rwandan context, as it was only modestly correlated with the YCPS-R and uburara diagnoses, and was not correlated with youth reported functional impairment. Therefore, the YCPS-R may be best used as a continuous measure of conduct problems rather than a diagnostic tool. Youth with elevated scores on the YCPS-R would be expected to have more functional impairment, but cannot be assumed to meet criteria for CD.
Some study limitations must be noted. The validity sample was a purposive sample, with approximately 67% of children referred for at least one mental health disorder, and the other 33% identified as not having any mental health disorder. The non-random sampling limits the generalizability of results. Additional studies are needed to examine the psychometric properties of the YCPS-R in a representative sample of Rwandan youth. While the YCPS-R was also developed for use with HIV-affected youth and their families, the uburara construct itself is expected to generalize to non-HIV affected youth. Additionally, it should be emphasized that the qualitative and quantitative results indicate that while uburara is related to CD and ODD, it is a distinct presentation of conduct problems in Rwanda, and not entirely equivalent to DSM-IV-TR CD or ODD criteria [36].
To our knowledge, this is the first measure developed in SSA for assessing youth conduct problems. The qualitative results indicated that the youth conduct problems in Rwanda are related to, but distinct from, conduct problem symptoms identified in western contexts, and that measuring them warrants the development of a new scale, rather than adaptation of an existing measure. The results lend support to the argument that the construct validity of scales must be assessed when the scales are used in new contexts, in addition to assessing the scale reliability and factor structure [41]. Moreover, this study provides an example of the importance, and process, of ensuring that measures are culturally relevant and appropriate for the settings in which they are used [42], [43].
Data Availability Statement
Given the sensitive nature of the data and the need to protect research participant confidentiality, the data used in this study has not been deposited in a public repository. However, the de-identified minimal data used to reach the conclusions and replicate the analyses described in this manuscript is available upon written request to, and after review and approval by, the Harvard School of Public Health's Office of Human Research Administration.
Acknowledgments
We would like to thank Ista Zahn, MA, of the FXB Center, for his statistical assistance.
Author Contributions
Conceived and designed the experiments: TSB. Performed the experiments: FK MM CM. Analyzed the data: LCN. Contributed reagents/materials/analysis tools: TSB LCN. Wrote the paper: LCN. Revised the article critically for important intellectual content: TSB FK MM CM. Approved final version to be published: TSB FK MM CM LCN.
References
- 1. Cortina MA, Sodha A, Fazel M, Ramchandani PG (2012) Prevalence of child mental health problems in sub-saharan africa: A systematic review. Arch Pediatr Adolesc Med 166: 276–281.
- 2.
Murray CJL, Lopez AD, editors (1996) The Global Burden of Disease and Injury Series, Volume 1: A Comprehensive Assessment of Mortality and Disability from Diseases, Injuries, and Risk Factors in 1990 and Projected to 2020. Cambridge, MA, USA: Harvard University Press.
- 3. Patel V, Flisher AJ, Nikapota A, Malhotra S (2008) Promoting child and adolescent mental health in low and middle income countries. Journal of Child Psychology & Psychiatry 49: 313–334.
- 4. Canino G, Polanczyk G, Bauermeister JJ, Rohde LA, Frick PJ (2010) Does the prevalence of CD and ODD vary across cultures? Soc Psychiatry Psychiatr Epidemiol 45: 695–704.
- 5. Kessler RC, Angermeyer M, Anthony JC, De Graaf R, Demyttenaere K, et al. (2007) Lifetime prevalence and age-of-onset distributions of mental disorders in the World Health Organization's World Mental Health Survey Initiative. World Psychiatry 6: 168–176.
- 6. Maughan B, Rowe R, Messer J, Goodman R, Meltzer H (2004) Conduct Disorder and Oppositional Defiant Disorder in a national sample: developmental epidemiology. Journal of Child Psychology and Psychiatry 45: 609–621.
- 7. Seedat S, Scott KM, Angermeyer MC, Berglund P, Bromet EJ, et al. (2009) Cross-national associations between gender and mental disorders in the WHO World Mental Health Surveys. Arch Gen Psychiatry 66: 785.
- 8. Tramontina S, Martins S, Michalowski MB, Ketzer CR, Eizirik M, et al. (2001) School dropout and conduct disorder in Brazilian elementary school students. Can J Psychiatry 46: 941–947.
- 9. Hinshaw SP (1992) Academic underachievement, attention deficits, and aggression: Comorbidity and implications for intervention. J Consult Clin Psychol 60: 893–903.
- 10. Whitmore EA, K. Mikulich S, Thompson LL, Riggs PD, Aarons GA, et al. (1997) Influences on adolescent substance dependence: conduct disorder, depression, attention deficit hyperactivity disorder, and gender. Drug Alcohol Depend 47: 87–97.
- 11. Windle M (1990) A longitudinal study of antisocial behaviors in early adolescence as predictors of late adolescent substance use: Gender and ethnic group differences. J Abnorm Psychol 99: 86–91.
- 12.
Boyle MH, Offord DR (1991) Psychiatric disorder and substance use in adolescence. The Canadian Journal of Psychiatry/La Revue canadienne de psychiatrie.
- 13. Kovacs M, Krol R, Voti L (1994) Early Onset Psychopathology and the Risk for Teenage Pregnancy among Clinically Referred Girls. J Am Acad Child Adolesc Psychiatry 33: 106–113.
- 14. Zoccolillo M, Rogers K (1991) Characteristics and Outcome of Hospitalized Adolescent Girls with Conduct Disorder. J Am Acad Child Adolesc Psychiatry 30: 973–981.
- 15. Brown LK, Hadley W, Stewart A, Lescano C, Whiteley L, et al. (2010) Psychiatric disorders and sexual risk among adolescents in mental health treatment. J Consult Clin Psychol 78: 590–597.
- 16. Chen X, Thrane L, Whitbeck LB, Johnson KD, Hoyt DR (2007) Onset of Conduct Disorder, Use of Delinquent Subsistence Strategies, and Street Victimization Among Homeless and Runaway Adolescents in the Midwest. J Interpers Violence 22: 1156–1183.
- 17. Angold A, Costello E, Farmer EM, Burns BJ, Erkanli A (1999) Impaired but undiagnosed. J Am Acad Child Adolesc Psychiatry 38: 129–137.
- 18. Wolff JC, Ollendick TH (2006) The comorbidity of conduct problems and depression in childhood and adolescence. Clin Child Fam Psychol Rev 9: 201–220.
- 19. Giaconia RM, Reinherz HZ, Silverman AB, Pakiz B, Frost AK, et al. (1995) Traumas and posttraumatic stress disorder in a community population of older adolescents. J Am Acad Child Adolesc Psychiatry 34: 1369–1380.
- 20.
Courtois CA (2008) Complex trauma, complex reactions: Assessment and treatment. Psychol Trauma S: 86.
- 21. Makame V, Ani C, Grantham-McGregor S (2002) Psychological well-being of orphans in Dar El Salaam, Tanzania. Acta Paediatr 91: 459–465.
- 22. Doku P (2009) Parental HIV/AIDS status and death, and children's psychological wellbeing. Int J Ment Health Syst 3: 26.
- 23. Murphy DA, Greenwell L, Mouttapa M, Brecht ML, Schuster MA (2006) Physical health of mothers with HIV/AIDS and the mental health of their children. J Dev Behav Pediatr 27: 386–395.
- 24. Lester P, Jane Rotheram-Borus M, Lee S-J, Comulada S, Cantwell S, et al. (2006) Rates and predictors of anxiety and depressive disorders in adolescents of parents with HIV. Vulnerable Child Youth Stud 1: 81–101.
- 25. Daud A, Skoglund E, Rydelius P (2005) Children and families of torture victims: Transgenerational transmission of parents' traumatic experiences to their children. Int J Soc Welf 14: 23–32.
- 26.
Ministry of Gender and Family Promotion (2008) A Situation Analysis of Orphans and Other Vulnerable Children in Rwanda. Kigali, Rwanda.
- 27.
UNAIDS (2012) Country Progress Report: Rwanda. Geneva, Swizerland.
- 28. Betancourt TS, Scorza P, Meyers-Ohki S, Mushashi C, Kayiteshonga Y, et al. (2012) Validating the Center for Epidemiological Studies Depression Scale for Children in Rwanda. J Am Acad Child Adolesc Psychiatry 51: 1284–1292.
- 29. Scorza P, Stevenson A, Canino G, Mushashi C, Kanyanganzi F, et al. (2013) Validation of the “World Health Organization Disability Assessment Schedule for Children, WHODAS-Child” in Rwanda. PLoS ONE 8: e57725.
- 30.
Betancourt TS, Rubin-Smith J, Beardslee WR, Stulac S, Fayida I, et al.. (2011) Understanding locally, culturally, and contextually relevant mental health problems among Rwandan children and adolescents affected by HIV/AIDS. AIDS Care iFirst: 1–12.
- 31.
Lee J (2012) Conducting cognitive interviews in cross-national settings. Assessment.
- 32.
SPSS (2009) PASW Statistics for Windows. 18.0 ed. Chicago.
- 33.
Sheehan D, Shytle D, Milo K, Janavs J, Lecrubier Y (2009) MINI International Neuropsychiatric Interview for Children and Adolescents, English Version 6.0.
- 34. Kovacs M, Paulauskas S, Gatsonis C, Richards C (1988) Depressive disorders in childhood III. A longitudinal study of comorbidity with and risk for conduct disorders. J Affect Disord 15: 205–217.
- 35.
StataCorp (2011) Stata Statistical Software: Release 12. 12 ed. College Station, TX: StataCorp LP.
- 36.
American Psychiatric Association (2000) Diagnostic and statistical manual of mental disorders (4th ed., text revision) Washington, DC: American Psychiatric Association.
- 37. Fendrich M, Weissman MM, Warner V (1990) Screening for Depressive Disorder in Children and Adolescents: Validating the Center for Epidemiological Studies Depression Scale for Children Am J Epidemiol. 131: 538–551.
- 38. Brislin RW (1970) Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology 1: 158–216.
- 39. Benjet C (2010) Childhood adversities of populations living in low-income countries: prevalence, characteristics, and mental health consequences. Current Opinion in Psychiatry 23: 356–362.
- 40. Hanley J, McNeil B (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29–36.
- 41. Jayawickreme N, Jayawickreme E, Atanasov P, Goonasekera MA, Foa EB (2012) Are culturally specific measures of trauma-related anxiety and depression needed? The case of Sri Lanka. Psychol Assess 24: 791.
- 42.
Kleinman A (1988) Rethinking psychiatry: From cultural category to personal experience. New York, NY: Free Press.
- 43. Bolton P (2001) Cross-cultural validity and reliability testing of a standard psychiatric assessment instrument without a gold standard. J Nerv Ment Dis 189: 238–242.