Comparing simulated client experiences with phone survey self-reports for measuring the quality of family planning counseling: The case of depot medroxyprogesterone acetate – subcutaneous (DMPA-SC) in Nigeria

Background

Global efforts to reduce unintended pregnancies and abortions and to increase utilization of contraceptive services include a range of programs and interventions specifically targeted towards improving quality of care^1,2. However, what constitutes quality care and how to measure it is complex. The Bruce-Jain framework (1994) for understanding the quality of contraceptive services included six domains: choice of method; information to client; technical competence; interpersonal relations; mechanisms to encourage continuity; and constellation of services³. Other frameworks have expanded upon this framework to include additional domains, such as access/accessibility and patient-centeredness^4,5.

Furthermore, a standard method for measuring quality in contraceptive services does not exist. Many measures primarily focus on technical aspects of quality that can be assessed objectively (e.g., facility stocks and infrastructure, provider competence)^1,6. Even studies that ask actual clients often rely on more objective questions, and struggle to capture their experiences of interpersonal or person-centered quality. Moreover, objective aspects of quality and clients’ subjective perceptions may not align⁷. Tumlinson et al. (2014) detail the advantages and disadvantages of commonly-used methods for measuring the quality of contraceptive services, including facility audits, direct observations, provider interviews, and client interviews, highlighting the fact that different methods yield different insight into different aspects of quality. For example, facility audits might identify issues related to stockouts, while direct observation may be useful for assessing provider competence. User interviews may be especially useful for assessing subjective aspects of quality, including satisfaction and client perceptions of quality. However, these methods are not without their limitations for measuring service quality, including courtesy bias, lack of reliability in self-reports, the Hawthorn effect, and recall bias.

Another methodological approach becoming increasingly common is the simulated client or “mystery client” method, wherein trained actors are sent to seek contraceptive services from a provider following a scripted interaction guide. These simulated clients then record the details of their interactions in a standard survey or interview format after they leave the facility. Ideally, this method allows for more standardization than asking actual clients or providers themselves as the actors can be trained to look for certain indicators of quality, and to have a standardized expectation of quality⁸. Recently, Tumlinson et al. (2014) used simulated clients to test the validity of client exit surveys and provider interviews (self-reported measures), and direct observations (conducted by a third party) in measuring quality; they found low specificity and positive predictive values of quality in all of these approaches. These results suggest that using different methodological approaches is necessary for understanding the complexity of women’s experiences of quality of care for contraceptive services, and that exploring the differences arising from such methods may yield new insights into how to improve both the clinical aspects and the patient-centeredness of care received.

Given these insights, we chose to use mixed methods—namely simulated clients and client self-reports via a follow-up phone survey—to assess the quality of services received by women seeking contraceptive services and particularly for obtaining DMPA-SC (depot medroxyprogesterone acetate – subcutaneous, also known as Sayana Press®), a new injectable contraceptive recently introduced into the Nigeria market. Furthermore, because this new delivery method for DMPA was specifically designed to reduce barriers to contraceptive uptake for underserved populations, including adolescent and young women⁹, we sought to additionally examine differences in contraceptive services experienced by unmarried women compared to married women. The overarching aim was to assess how the quality of service provision could be improved in order to encourage greater uptake and method continuation.

Despite recent increases, contraceptive use is generally low in Nigeria, reaching 11.1% in 2013 for modern methods¹⁰. Injectables are increasingly popular, and comprise the largest increase in the contraceptive prevalence rate among married women in the past three years¹¹. However, these gains are not equal. Although unmarried Nigerian women are more likely to use modern contraception, they are also at increased risk for unintended pregnancies, suggesting that consistency in use of effective methods remains poor^12,13. In general, adolescents and young adults have substantial unmet need for sexual and reproductive health services and related information¹⁴ and many avoid seeking services due to poor quality of care.

Unlike other countries where DMPA-SC has been introduced and studied as small pilot projects^15–17, DMPA-SC was introduced in Nigeria in 2015 via a mass market approach. Distribution occurred through multiple channels—facilities, drug shops, and specially-trained community-based distributors—in the private health care sector which accounts for the majority of all health services provided, and in particular for contraceptive services¹⁰. This presented an opportunity to assess actual user experiences with DMPA-SC from the broader population of new and continuing contraception users under real world conditions. Our data collection focused on DMPA-SC service provision from selected private sector providers across seven states in South West Nigeria (where the introductory efforts were concentrated) in 2016. In order to more fully understand the experiences of married and unmarried users, we chose to use multiple methods to assess various aspects of quality of service provision. In this paper, we directly compared the quality of care along specific measures collected from both the simulated clients sent to providers selling DMPA-SC and the follow-up phone surveys with women who recently obtained DMPA-SC at the same facility. To understand whether these varied methodological approaches could identify the same differences in the quality of care among population sub-groups, we further analyzed the responses by marital status.

Methods

Results

The phone survey included 541 women who had visited a private facility for DMPA-SC and were using it (see Underlying data¹⁸). As shown in Table 1, almost two-thirds (65.8%) of phone survey participants visited a community-based distributor to obtain DMPA-SC, followed by private hospital/clinic/maternity homes (11.1%), and government hospital/clinic/maternity home (private providers employed at public facilities) (11.7%). About 6% of women attended a pharmacy, and 3.7% attended a retail drug shop. Few women attended private doctors or nurses. The majority of women were currently married (93.0%) and age 25 or older (only 9% were under 25). Most switched from another modern method, with just over one-quarter being new users of contraception. Simulated clients completed visits for both profiles at a total of 122 facilities (Table 2). The majority of these facilities or providers were located in urban areas (72.1%), with most of the remainder in peri-urban areas (24.6%). Less than half of providers visited were female (42.6%).

Discussion

To assess the quality of contraceptive counseling in the provision of DMPA-SC delivered through private sector providers in South West Nigeria, we compared measures of technical quality from users’ experiences self-reported in a follow-up phone survey and with simulated client visits. Our findings suggest that different methodological approaches can yield similar results but with varying intensity about the quality of care. Both methods were able to detect systematic differences in quality for women of different marital statuses. Across all measures, phone survey respondents perceived higher levels of quality than simulated clients, however, the direction of the difference in quality of care between young unmarried clients and older married clients was similar: younger unmarried women received lower quality of care for contraceptive services. In sensitivity analyses, we restricted the analyses to the same set of providers who saw both simulated clients and phone survey respondents; responses were similar in magnitude and direction.

These findings support previous research suggesting that asking women directly about their experiences can lead to higher quality scores than more objective approaches (such as simulated clients), even for aspects of technical quality that may be less open to subjective rating and only require a yes/no response. While actual users’ responses may better capture their perceived experiences, these results may be less useful to program implementers because they are determined by the socio-cultural expectations governing such interactions, which in this case, are reflective of a population that may have low expectations of quality of care. In contrast, a key advantage of the simulated client approach is the standardization of perspectives on quality⁸, which ultimately yield greater variation in quality measurements with which to identify where improvements in service provision are needed.

In fact, our findings coupled with our experiences in training our simulated client actors suggest that self-reported user perceptions may reflect more reflexive reactions or reactions based on social expectation, rather than a more detached, independent assessment of quality that researchers or program implementers intend. Many of our simulated client actors similarly perceived higher levels of quality in role play during initial training sessions than after eventual calibration. When mock patient-provider interactions were analyzed more systematically to standardize ratings, actors’ perceptions of quality declined, particularly among unmarried profile actors. Compared to the older women actors assuming the married profile, our younger women actors did not seem to expect to receive higher quality care, and thus initially rated the mock interactions to be higher quality than older women for the same observed encounter. Thus, questions about the quality of clinical care and counseling may be less salient for unmarried women seeking contraceptive services who do not expect to receive higher quality of care or who may have few reference points with which to assess relative quality of care. Other studies have found that younger contraceptive users in Nigeria have limited contact with contraceptive services¹⁴. However, when expectations of quality are standardized across individuals, unmarried women may be more likely to recognize relatively worse quality of care, potentially explaining the larger magnitude of differences by marital status found with simulated clients as compared to phone survey respondents. Thus, users’ assessments of the quality contraceptive service provision should be interpreted in light of prevailing social and cultural expectations, which may differ across subgroups.

Additionally, women’s expectations of quality may not necessarily align with programmatic or international standards of quality. While our results suggest that user self-reports may provide a better summary depiction of their lived experiences, it may behoove researchers to question if the common aspects of quality adopted by researchers and practitioners are salient to the women represented by the sample. While this study was only able to directly compare quality measures for a limited subset of technical competency measures purposively chosen for that purpose, the relevant dimensions of quality for a particular population may be highly context-dependent. For example, recent research on person-centered quality of care for childbirth in Kenya found that even rural and urban populations of the same country identified different important indicators for quality¹⁸.

These differences in quality ratings between methods were similar to those found by Tumlinson et al. between their simulated clients and user self-reports in exit interviews⁶. Thus, even though our phone surveys were conducted with some delay after the service was rendered rather than immediately afterward vis-a-vis exit interviews, recall bias for more technical quality measures did not appear to affect overall user ratings.

Several study limitations should be noted. First, while simulated clients went to the same facilities as phone survey respondents, this does not mean that they saw the same provider. This is most likely especially true for the larger facilities, but also the pharmacies. Thus, the comparison between these two methods may not be reflective of interactions with the same provider, only the same facility. Phone survey interviews with actual users were conducted within a few weeks of the provider interaction, which may introduce recall bias, unlike with our simulated client data obtained from clients debriefed within 30 minutes of their interaction with providers. Although simulated client interactions and profiles were standardized across actors as much as possible, providers may still perceive actors to be members of better educated and wealthier classes given their mannerisms and speech. Due to the large geographic area covered by the DMPA-SC program under study, local actors from targeted communities could not be recruited for all selected localities. Respondents for the phone survey were disproportionately drawn from clients of community-based distributors, who were more likely to provide services to unmarried women, than other provider types. Yet, most of our results still hold when the phone survey sample is restricted to the same set of providers included in the simulated client interactions, although this sample size is too small to make any claims of significance.

Conclusions

Measuring the quality of contraceptive provision is essential for improving the experiences of clients, and subsequently increasing the uptake and continuation of contraceptive use. However, measuring the quality of contraceptive services has proven to be difficult as different approaches have specific limitations and advantages. Our study has compared two different approaches to measuring quality of contraceptive services in the private healthcare sector. Both approaches confirmed that unmarried clients receive lower quality of care compared to married clients. The simulated client method is more time- and resource-intensive than exit or follow-up interviews. However, this method has the potential to provide a more standardized perspective on the quality of the provider-client interaction, while still collecting valuable information on important differences in user experiences across subgroups. It is not affected by recall bias unlike phone interviews conducted several weeks after the interaction. It can also allow for better control of the effects of social expectations unlike all other direct interviews of actual clients, including client exit. Depending on the research and intervention agenda, having standardized, comparable data might be most beneficial, for example, when comparing programs or across populations. On the other hand, other research might benefit more from understanding women’s lived experiences of provider interactions and quality, as well as their expectations and perceptions of specific aspects of quality, for example when aiming to improve a specific program. These findings can help future researchers select the most appropriate methodological approach for their research goals.

Data availability