Assessing the reach and effectiveness of mHealth: evidence from a reproductive health program for adolescent girls in Ghana

Abstract

Background

While mobile health (mHealth) programs are increasingly used to provide health information and deliver interventions, little is known regarding the relative reach and effectiveness of these programs across sociodemographic characteristics. We use data from a recent trial of a text-messaging intervention on adolescent sexual and reproductive health (SRH) to assess the degree to which mHealth programs reach target adolescent subpopulations who may be at higher risk of poor SRH outcomes.

Methods

The study was conducted among girls aged 14–24 in 22 secondary schools in Accra, Ghana. The mHealth intervention was an interactive mobile phone quiz in which participants could win phone credit for texting correct answers to SRH questions. We use detailed data on individuals’ level of engagement with the program, SRH knowledge scores, and self-reported pregnancy collected as part of the original trial to assess the extent to which engagement and program impact vary across parental education, sexual experience, SRH knowledge deficit, and parental support.

Results

Eighty-one percent of participants engaged with the mHealth program, with no evidence that the program disproportionally reached better-off groups. The program was effective at increasing knowledge of SRH across all strata. Higher levels of engagement were associated with higher knowledge scores up to year later. There was no significant impact of the program on self-reported pregnancy within subgroups.

Conclusion

mHealth programs for adolescents have the potential to engage and increase SRH knowledge of adolescent girls across sociodemographic strata, including those who may be at higher risk of poor SRH outcomes.

Trial registration

ClinicalTrials.gov NCT02031575. Registered 07 Jan 2014.

Background

With the rapid expansion of mobile phone ownership in low- and middle-income countries (LMICs) over the past decade, the field of mobile health (mHealth) has emerged as a novel and potentially cost-effective way to increase access to health information and improve health knowledge as well as health outcomes [1,2,3,4].

mHealth programs have been particularly popular in the area of adolescent sexual and reproductive health (SRH), where knowledge gaps remain large [5,6,7,8,9,10,11]. A recent review of evidence and progress over the past 20 years in adolescent SRH highlights the persistent lack of access to SRH information and services, as well as the large knowledge gaps with respect to use of contraception [12].

A growing body of evidence highlights early sexual debut and one-parent households as key risk factors for adolescent pregnancy and childbearing, and knowledge about contraception as a key protective factor against these outcomes [13]. Structural influences such as low parental socioeconomic status, low parental education and lack of parental support with regards to SRH have also been associated with adolescent risky sexual behaviour and inconsistent contraceptive use in the literature [13,14,15].

Recent evidence suggests that mHealth programs can increase average health knowledge and that these programs are generally well-received by youth [5,6,7]. However, very little is known regarding the reach and effectiveness of these programs in populations with key risk factors [16]. While mobile phone ownership and usage has greatly expanded among young people in many LMIC settings [17], adolescents from disadvantaged groups generally have reduced rates of phone ownership or may only have limited access to a shared household phone [18,19,20,21]. In addition, youth from these key populations may face other barriers pertinent to mHealth such as decreased technological literacy, inferior network coverage, and lower linguistic competency [2, 3].

In this study, we analyse detailed data collected as part of a randomized trial of a text-messaging intervention conducted among secondary school students in Accra, Ghana. As part of the trial, adolescents were invited to participate in an interactive mobile phone program aimed to increase knowledge of SRH. We seek to answer the following research questions: (1) how effective was the program in engaging adolescents from key target populations including adolescents with low parental education, early sexual debut, low parental support, and low knowledge of SRH; (2) how did the impact of the program on SRH knowledge vary with level of engagement; and (3) how successful was the program at improving SRH knowledge and reducing incidence of pregnancy in key target populations?

Methods

Study setting

The study was conducted in the urban area of Accra, Ghana [1]. In a survey from 2015 in Ghana, 31% of respondents aged 14–18 years and 71% of respondents aged 19–25 years owned a mobile phone, while 77% of those aged 14–18 and 91% of those aged 19–25 used a mobile phone in the past 4 weeks [20].

Despite enacting a National Adolescent Reproductive Health Policy in 1996, the SRH of Ghanaian adolescents remains a significant challenge. According to the 2014 Demographic and Health Survey, by age 19, 36% of women have given birth or are pregnant [22]. Among adolescent mothers under 20 years of age, 58% had not planned on having the pregnancy at that time. Use of contraception among adolescents is low: among sexually active unmarried adolescents aged 15–19, less than a third use a modern method of contraception [22].

Though adolescents are generally aware of HIV and pregnancy risks, SRH knowledge tends to be limited and mostly superficial. Results of a 2004 national survey of Ghanaian adolescents found that while more than 95% of girls aged 15–19 knew of at least one method of modern contraception, only 67% were aware that a woman is more likely to get pregnant on certain days than others. Only 28% knew both previous items and were able to reject two common misconceptions (that a woman cannot get pregnant the first time she has sex or if she has sex standing up) [23]. Moreover, widespread misconceptions about the correct way to use contraception and fears about long-term side effects on future fertility add to the barriers to modern contraception use among adolescents [24,25,26].

Procedures

We used secondary data from a recent randomized trial that provided SRH information by text message (also known as short message service, or SMS) to adolescent girls in Ghana. The original trial, which is described in further detail in Rokicki et al. (2016), found relatively large and persistent increases in SRH knowledge among adolescents participating in the interactive mHealth program [5].

The sampling frame for the study was provided by the 2012–2013 Ghana Education Service Register of Secondary Schools in Greater Accra. We randomized 38 schools to the interactive mHealth intervention (n = 12), a simplified unidirectional messaging intervention (n = 12), and the control group (n = 14). After randomization, we found 3 schools to be ineligible and 1 refused to participate due to time constraints [5]. In this paper, we focus only on the interactive mHealth intervention arm and the control arm. The analytic sample consisted of 10 schools assigned to the intervention (N = 205) and 12 schools assigned to control (N = 293).

We blocked randomization by school category (a measure of quality designated by the Ghana Education Service) and by whether the school had a home economics class. We chose classes within schools based on maximizing the number of eligible girls; if a home economics class was offered in the school it was chosen because it was always a large class of nearly all female students. Home economics was not offered at all schools so stratifying ensured even distribution across arms.

The inclusion criteria for schools was senior high day schools in the Greater Accra region; boarding schools were excluded. Within schools, sampling was restricted to female students, aged 14–24 years, in one class in the second year of senior high school. Participants used their own mobile phones or could use a family member’s phone; no phones were provided.

Participants gave written consent; those aged younger than 18 years obtained parental consent. Institutional Review Board (IRB) approval was granted by Harvard University [#FWA00004837] as well as locally by the Ghana Health Service (GHS-ERC:05/09/13). The study design was registered on ClinicalTrials.gov (NCT02031575).

We visited schools at baseline, immediately after the intervention was completed (3-month follow-up), and 1 year later (15-month follow-up). At each visit, participants completed a self-administered questionnaire. Participants’ demographic information was completed at baseline. Reproductive health knowledge was assessed at each time-point. Information on sexual behavior and pregnancies was collected only at the 15-month follow-up.

The mHealth platform

The mHealth platform was designed as an interactive mobile phone quiz game in which participants could win airtime (i.e. mobile phone credit that can be used for making calls or sending texts) for texting correct answers to SRH questions. To design the message content, we first conducted focus groups with adolescents to understand their most prevalent SRH concerns, followed by a round table discussion with the Health Promotion Unit at Ghana Health Service who approved appropriateness of topics and finalized wording.

For a period of 12 weeks, participants were sent one multiple-choice quiz question about SRH each week via text message to which they were invited to respond free of charge. These messages focused on pregnancy prevention and contained information on topics of reproductive anatomy, pregnancy, sexually transmitted infections (STIs), and contraception including male and female condoms, birth control pills, and emergency contraception. Upon responding, participants immediately received a confirmatory text message informing them whether they answered correctly, the correct answer, and additional information. Participants were sent up to 2 reminder messages if they did not respond; those who had not responded by the end of the week were sent a text message with the correct answer and the additional information at the end of the week. Participants in the same school were encouraged to discuss messages with each other. Participants were told that correct answers were rewarded: for every 2 correct responses, participants were sent 1 GHS (US$0.38) of airtime credit at the end of the week.

Over the same 12-week intervention period, the control group participants were sent one message each week with information about malaria. The content of all messages is shown in Table 4 in Appendix. Participants in the control group were interviewed at baseline, 3 months, and 15 months using the same procedures as participants in the intervention group. All messages were sent in English, the official language of instruction in all secondary schools in the country, using secure servers via Telerivet service; non-delivered messages were re-sent.

Measures

To measure program engagement, we used two separate variables. First, we measured the total number of times the respondent replied to the weekly text-message quiz questions (maximum of 12), from data extracted from the mobile phone records. Second, we measured self-reported message exposure. At the 3-month follow up, respondents were asked “How often did you receive messages from [the program]” [More than once a week/ About once a week/ About once a month/ Less than once a month]. We created an indicator for having received messages at least once a week.

To evaluate reproductive health knowledge, participants completed a true-or-false test consisting of 24 items (see Table 5 in Appendix for details). Items on the test were adapted from the Guttmacher Institute’s 2009 National Survey of Reproductive and Contraceptive Knowledge for the setting of Ghana [27]. Knowledge scores were calculated as the percentage of items answered correctly; we then calculated knowledge z-scores by subtracting from each score the overall mean at baseline and dividing by the standard deviation.

In calculating knowledge scores, “don’t know” answers and missing values were treated as incorrect answers. The percentage missing for each item was low and is shown in Table 5 in Appendix. At the 15-month follow-up, data collection was done using tablet computers so there were no missing values in those scores. For the baseline and the 3-month follow-up, which were done on paper questionnaires, we re-calculated knowledge scores such that items with missing values were excluded from the calculation (that is, the knowledge score was calculated as the percentage correct of the total number of items answered). The correlation coefficients between treating missing values as incorrect and excluding missing values from the calculation was 0.92 at baseline and 0.99 at the first follow-up.

Self-reported pregnancy was assessed at the 15-month follow-up with the question “In the past year, have you been pregnant?” [Yes/ No/ I don’t want to answer].

Our explanatory variables were selected ex-ante on the basis of theory and prior evidence of risk and protective factors for adolescent SRH [13]. We identified these as: 1) adolescents from households of low socioeconomic status (defined as both parents completed only primary school or less), 2) adolescents who were sexually active at baseline, 3) adolescents who have a larger than average SRH knowledge deficit (earned a baseline knowledge z-score of less than 0), and 4) adolescents who have low parental support (do not strongly agree or agree a little bit with the statement “I feel comfortable talking to my parents about condoms and contraception” [Strongly agree, Agree a little bit, Neither agree nor disagree, Disagree a little bit, Strongly disagree]).

Statistical analysis

Our analysis is divided in three parts. First, we evaluated characteristics associated with engagement with the interactive mHealth program. We used a Poisson regression model to examine the association between number of responses to the text-messsage quiz questions as our outcome and parental education, sexual experience, parental support, and SRH knowledge as explanatory variables. We also verified similarity of results to a negative binomial model to account for over-dispersion (Table 6 in Appendix). Additionally, we used logistic regressions to examine the associations of these characteristics with a binary variable for any response to the text-messages, as well as with a binary variable measuring participation as whether the participant self-reported to have received messages at least once a week during the course of the program.

Next, to evaluate the impact of engaging with the program on reproductive health knowledge, we used linear regression models of knowledge z-score at both 3 and 15 months as a function of level of engagement (number of responses) of the intervention group. We used quantile-quantile plots to assess normality of the residuals.

Finally, we assessed variation in program impact across target subgroups. We used linear regression models stratified by subgroup to estimate the impact of the intervention on knowledge z-score. We then used models with an interaction term for intervention group and subgroup indicator to test equality of the coefficients. For these interaction tests, marginal effects were derived from the interaction model, and then compared using linear hypothesis tests.

Due to the small cell sizes in subgroups, we used exact logistic regression models, again stratified by subgroup, to estimate the impact of the intervention on self-reported pregnancy in the past year.

Missing values and item response refusals were included in the analyses as separate categories; results for these categories are shown in Tables 6 and 7 in Appendix. We adjusted all models for blocking variables, which were category of school (higher quality public, lower quality public, and private) and an indicator for the presence of home economics class) and we clustered standard errors at the school level to correct for within-school correlation of outcomes [28, 29]. We used Stata v14 for all analyses [30].

Results

Table 1 shows the baseline characteristics of the participants, which were similar for intervention and control groups. At baseline, 58% of participants were 17–18 years of age, while 18% were 16 and under and 24% were 19 and over. Almost 70% of participants were not sexually active at baseline while 13% were sexually active; another 13% refused to answer the question. 8% of participants reported that both parents had low education (primary or less), 23% reported that at least one parent had higher education, 49% reported that both parents had higher education, while 20% did not know. About a third of participants (32%) reported that they had high parental support, while the rest reported low parental support.

Table 1 Baseline risk profile and socioeconomic characteristics (n (%))

Differences in program engagement by target subgroups

Table 2 shows the results of the Poisson and logistic regression models investigating characteristics associated with engagement with the mHealth program. Overall, responsiveness to the program was high: 81% responded via text message to at least one quiz question; the average number of responses was 8. The histogram of responses is shown in Figure 3 in Appendix. Adjusting for other covariates, those who reported that both parents had low education had a significantly higher rate of responding to messages (Incidence Rate Ratio (IRR) 1.22, 95% CI 1.03–1.46), but were not more likely to report receiving messages. We found no difference in program engagement by baseline sexual activity, knowledge, or parental support.

Table 2 Characteristics associated with program engagement as measured by number of responses, response to any messages, and self-reported message receipt (intervention group only)

Association between engagement and knowledge

Figure 1 shows the difference in average knowledge z-score between intervention and control groups as a function of the intervention group's engagement with the program (as measured by the number of responses). Higher levels of engagement were associated with higher knowledge scores both at 3 months (linear slope estimate 0.11, 95%CI 0.08 to 0.14) and at 15 months (linear slope estimate 0.07, 95%CI 0.02 to 0.13).

Fig. 1

Average difference in knowledge z-score between intervention and control groups, at 3 months (left) and 15 months (right), as a function of number of responses to text message quiz questions (uses intervention and control groups, N = 498)

Difference in program impact on high risk groups

Figure 2 shows the difference in knowledge z-score for the intervention compared to the control, stratified by target subgroups. The mHealth program was effective at significantly increasing knowledge for every subgroup at both 3 and 15 months. We found no heterogeneity in the effect of the intervention across any subgroup population (Table 7 in Appendix).

Fig. 2

Difference in knowledge z-score for intervention group compared to the control group at 3 and 15 months, stratified by target subgroups (uses intervention and control groups, N = 498)

Finally, Table 3 shows the impact of the mHealth program on self-reported pregnancy in the past year (reported at 15 months), stratified by target subgroups. Among those sexually active at baseline, 2 of 25 participants (8%) in the intervention group reported a pregnancy, while 6 of 39 participants (15.4%) in the control group reported a pregnancy. However, the odds ratio estimate was not significant and confidence intervals were wide due to the small cell sizes in these subgroups.

Table 3 Relative program impact on self-reported pregnancy in the past year for target subgroups (uses intervention and control groups, N = 498)

Discussion

The results presented in this paper suggest that mHealth programs are not only an effective tool to increase SRH knowledge overall in the studied setting, but that these programs can also engage and increase SRH knowledge of adolescents from key target populations who are at higher risk of poor SRH outcomes, including adolescents with low parental education, adolescents with low SRH knowledge, adolescents with early sexual debut, and adolescents with low parental support.

We found that overall engagement in the program was high, and that participants in target groups were just as likely to interact with the program as their counterparts. In fact, participants whose parents had low levels of education had a higher rate of responding to messages than those whose parents had high levels of education.

Program engagement also appears to be important for knowledge absorption. We found that those who interacted more with the program had significantly greater knowledge of reproductive health in both the short-term (immediately after the program ended) and in the longer-term (a year later), compared to those who did not interact. Those who did not reply to messages still received the reproductive health information at the end of each week; however, knowledge improvements were minimal at 3 months and not significantly different from the control at 15 months. We cannot interpret the differences causally, as there may be confounding factors such as motivation that affect both desire to interact with the program and knowledge. However, the findings point to the potential importance of including features such as interaction and feedback in the design of mHealth programs for adolescents [3, 31].

We also found that the mHealth program was effective at increasing SRH knowledge across levels of parental education, SRH knowledge, sexual experience, and parental support, in both the short-term and long-term, a surprising finding considering the number of potential cultural and technological barriers that participants in target groups may encounter.

Finally, we found no significant impact of the mHealth program on self-reported pregnancy within subgroups. Although girls in the intervention group who were sexually active at baseline had fewer self-reported pregnancies than their counterparts in the control group, the difference was not significant and sample sizes were too small to draw strong conclusions. Providing information via mobile phone to the population that needs the information most – in this case, girls who are sexually active and who immediately benefit from information about contraception and pregnancy prevention – may be an effective way of reducing unintended pregnancy rates; however, the study analyzed here was not powered to directly address this question, and more research is needed to understand the effect of mHealth programs on unintended pregnancy in sexually active adolescent populations.

This analysis has a number of limitations. First, there may be measurement error in some of the outcomes studied. Sexual activity and pregnancy were self-reported and thus may be subject to recall bias and social desirability bias. The bias may be differential by intervention group if, for example, provision of sexual health information by mobile phone made participants more likely than those in the control group to respond accurately and honestly to SRH questions. It may also be differential by target subgroup; for example, those with stronger parental connectedness or more baseline SRH knowledge may also respond more honestly. However, neither the size nor the direction of such biases is obvious. In other studies, adolescents’ self-reported sexual behavior data has been reported to suffer from inconsistencies, recall error, and misunderstanding or confusion surrounding the survey question [32]. In future studies, biological markers of pregnancy would be ideal to fully understand the health impact of the program. Another data limitation of the study presented is that a large proportion of girls (20%) did not know the level of their parent’s education. It is also worth noting that participants were recruited from secondary school in an urban area and thus are not representative of the general population; heterogeneity in terms of socioeconomic and demographic characteristics in the general population may be much larger. In particular, girls from rural areas and girls who stopped schooling before secondary school may have less access to mobile phones or reduced literacy skills than the girls in our sample. As this study was a small scale pilot study, we did not include boys. In terms of interpreting impacts, the intervention was a multi-component program that included participant interaction with quiz questions, reminder messages, and rewards; we are not able to differentiate impact as a result of individual components. Finally, only about a third of participants had ever had sex at the 15-month follow-up resulting in small sample sizes; longer-term impact of the program once a majority of girls become sexually active is unknown.

Our analysis also has a number of strengths. The original trial was a randomized design, and thus reduces the risk of confounding biases relative to observational data. In addition, follow-up rates at both 3 and 15 months were over 95% [5].

Previous research has found that boys are more likely than girls to use mobile phones, and youth from higher socioeconomic backgrounds are also more likely to own and access mobile phones [18,19,20]. An evaluation of a district-wide HIV/AIDS mHealth campaign in Uganda found that women and those with lower SRH knowledge were less likely to respond to SMS quiz questions than men and those with higher SRH knowledge [9, 33]. Conversely, our findings show that girls from across socioeconomic strata and levels of SRH knowledge do actively engage in such programs when they are directly targeted, and when intervention content is tailored to their needs. For future interventions, developing such customized content is undoubtedly of critical importance, and will likely require additional qualitative work of a wider range of age groups and geographic areas. From a policy perspective, it will also be important to use additional approaches to enrol socially disadvantaged girls into the program, since many may not pursue secondary education. Actively recruiting girls from disadvantaged neighborhoods for mHealth programs, rather than promoting programs through general advertising, may be an effective way to reach these vulnerable populations.

Conclusions

Adolescent girls in LMICs face substantial sexual and reproductive health risks, including unintended pregnancy, unsafe abortion, and HIV infection [34]. Accessing SRH information and services is challenging for adolescents due to a large number of barriers, including a lack of availability of accurate, comprehensive, and timely information [12]. As access to mobile phones among young people continues to rapidly expand, implementing mHealth programs in the area of adolescent SRH is becoming popular among academics and policymakers [5,6,7,8,9,10,11]. At the same time, concerns have grown that mHealth programs are not reaching vulnerable populations [18, 21, 35]. Our results suggest that mHealth platforms for adolescents have the potential to engage and increase health knowledge of adolescent girls across sociodemographic strata, including those who may be at higher risk of poor SRH outcomes.

Appendix

Table 4 Text message content for Intervention and Control groups

Table 5 Knowledge Test items

Table 6 Comparison of models for factors associated with number of responses, and full results for outcomes of responded to any message and self-reported message receipt

Table 7 Linear regression results for impact of mHealth treatment on sexual and reproductive health knowledge z-score, stratified by target subgroups, at 3-month and 15-month follow-ups

Fig. 3

Distribution of number of responses