Figures
Abstract
Background
The vulnerability of clinical trials to volunteer bias is under-reported. Volunteer bias is systematic error due to differences between those who choose to participate in studies and those who do not.
Methods and Results
This paper extends the applications of the concept of volunteer bias by using data from a trial of probiotic supplementation for childhood atopy in healthy dyads to explore 1) differences between a) trial participants and aggregated data from publicly available databases b) participants and non-participants as the trial progressed 2) impact on trial findings of weighting data according to deprivation (Townsend) fifths in the sample and target populations. 1) a) Recruits (n = 454) were less deprived than the target population, matched for area of residence and delivery dates (n = 6,893) (mean [SD] deprivation scores 0.09[4.21] and 0.79[4.08], t = 3.44, df = 511, p<0.001). b) i)As the trial progressed, representation of the most deprived decreased. These participants and smokers were less likely to be retained at 6 months (n = 430[95%]) (OR 0.29,0.13–0.67 and 0.20,0.09–0.46), and 2 years (n = 380[84%]) (aOR 0.68,0.50–0.93 and 0.55,0.28–1.09), and consent to infant blood sample donation (n = 220[48%]) (aOR 0.72,0.57–0.92 and 0.43,0.22–0.83). ii)Mothers interested in probiotics or research or reporting infants’ adverse events or rashes were more likely to attend research clinics and consent to skin-prick testing. Mothers participating to help children were more likely to consent to infant blood sample donation. 2) In one trial outcome, atopic eczema, the intervention had a positive effect only in the over-represented, least deprived group. Here, data weighting attenuated risk reduction from 6.9%(0.9–13.1%) to 4.6%(−1.4–+10.5%), and OR from 0.40(0.18–0.91) to 0.56(0.26–1.21). Other findings were unchanged.
Citation: Jordan S, Watkins A, Storey M, Allen SJ, Brooks CJ, Garaiova I, et al. (2013) Volunteer Bias in Recruitment, Retention, and Blood Sample Donation in a Randomised Controlled Trial Involving Mothers and Their Children at Six Months and Two Years: A Longitudinal Analysis. PLoS ONE 8(7): e67912. https://doi.org/10.1371/journal.pone.0067912
Editor: Robert K. Hills, Cardiff University, United Kingdom
Received: December 21, 2012; Accepted: May 22, 2013; Published: July 9, 2013
Copyright: © 2013 Jordan et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The project was supported by the Knowledge Exploitation Fund, Collaborative Industrial Research (project number HE09 COL 1002), Welsh Development Agency, United Kingdom (UK) and Cultech Limited, Unit 3, Christchurch Rd, Baglan Industrial Park, Port Talbot, SA12 7BZ, UK. Cultech Ltd. UK part funded the trial and provided the probiotic and matching placebo, and generated the random allocation sequence. Sue Plummer is a Director of Cultech and advised on study design and contributed to the final report. The sponsors were not involved in data collection, analysis or interpretation of the findings. The Knowledge Exploitation Fund had no involvement in the conduct of the trial. The funders had no role in data collection and analysis, decision to publish, or preparation of the manuscript. Dr. Sue Plummer proposed the product to be researched and commented on the study design. Dr. Sue Plummer and Dr. Iveta Garaiova commented on the manuscript prepared by Sue Jordan to ensure clarity. No changes to the findings were either suggested or made.
Competing interests: S. Jordan, S. J. Allen, C. A. Thornton, A. Watkins, R. Jones, I. Russell, C. J. Brooks, M. L. Heaven, and G. Morgan declare no conflicts of interest. M. Storey received financial support from Cultech Ltd. I. Garaiova is a Senior Research Manager and S. F. Plummer is the Managing Director, Obsidian Research Ltd. The affiliations of two authors and the commercial funding, along with any other relevant declarations relating to employment, products in development does not alter the authors’ adherence to all the PLOS ONE policies on sharing data and materials, as detailed in the online guide for authors. The product in the trial was not commercially available at the time of the trial.
Introduction
Recruitment to trials is deteriorating, particularly in developed countries [1]: industry sources suggest that recruitment rates across all trials fell by 75% between 1999–2002 and 2003–2006 [2]. Similarly, the proportion of recruits withdrawing from trials steadily increased between 1955 and 2000 [3], prompting the US Food and Drug Administration (FDA) to insist on measures to minimise missing data and the consequent bias [4].
Bias arising within trials, due to systematic differences between trial arms, threatens internal validity [5]. In addition, the external validity, generalisability, transferability and utility of well-conducted trials may be threatened where recruited and retained samples are less than 100% of the target population. Such selection bias, defined as the introduction of error due to systematic differences in the characteristics between those selected and those not selected for a given study [6], renders the recruited sample unrepresentative of the target population [7]–[9].
Recruitment of volunteers is a potential source of selection bias [10]. Where a sample can contain only those willing to participate in the study or experiment, systematic differences may arise between those who volunteer and those who decline or do not respond to invitations. Such “volunteer bias” is defined as any process at any stage of inference which tends to produce results or conclusions that differ systematically from the truth, arising where volunteers from a specified sample may exhibit exposures or outcomes which differ from those of non-volunteers [11]. Volunteer bias may arise during recruitment, retention, participation in follow-up clinics [12], and consent to blood sample donation. Volunteers may differ from the target population not only in socio-demographic characteristics, but also in less tangible ways, such as perceptions of the study’s leverage, saliency or relevance [13], or altruism [14].
The antithesis of volunteer bias, non-response bias [11], has been well scrutinised in surveys [13], [15] and observational studies requiring consent [16]. Although trials suffer higher non-response rates than surveys or observational studies [17]–[19], analysis of trial data rarely accounts for volunteer bias [7], [20]. Searches of three databases (PubMed, Web of Science, Scopus) located no reports of predictors of participation and consent to sample donation by well infants in clinical trials, using the key word/MeSH term combination: randomised controlled trials, pregnant women, infants, preventive therapy, research subject recruitment, loss to follow up, non-response bias, with or without “blood specimen collection”.
Little is known about families who decline to participate in clinical trials [21]. While there are exceptions, such as parents of seriously ill children [14], and situations where research offers the only access to free medication [22], non-targeted recruitment in all research designs favours healthier, wealthier, better educated, non-smokers, risking volunteer bias [10], [12], [17]–[19], [23]–[27]. The potential consequences of volunteer bias might be summarised [28]:
- Volunteer bias threatens the generalisability or external validity, transferability, and utility of findings and detracts from their clinical value [20]. When ‘hard to reach’ sections of the population are not included in a study, there can be no certainty that findings will be applicable to them. If a trial has been conducted in a population judged to be over-restricted, dissimilar or unrepresentative, findings may be dismissed as irrelevant. Prevention or vaccine trials are particularly vulnerable to such criticisms [7], [19], [26], [29]–[33].
- The incidence of disease in the recruited sample may be lower than accounted for in sample size calculations based on the incidence of disease in the whole population. This could leave the trial under-powered even when the target sample size has been recruited.
- Where trials report on conditions whose prevalence varies across the socio-demographic spectrum, findings, particularly estimates of the absolute effects of interventions (such as numbers needed to treat or harm, and costs), are often affected by over- or under-representation or exclusion of certain groups [7].
Evidence on which to base practice recommendations for wide sections of the population requires ‘Research evidence reflecting the diversity of the population’ [34], and trials with minimal demographic imbalance in recruitment and retention [28]. This paper aims to extend the application of the concept of volunteer bias to clinical trials, using data from a paediatric trial, by exploring:
- Potential for Volunteer Bias
- Differences between the recruited sample and the target population.
- Impact on retention, clinic attendance, consent to skin-prick testing and blood sample donation by well infants of i) demographics ii) leverage, saliency and altruism.
- Adjustment for potential volunteer bias by weighting outcome data [8] according to material deprivation (Townsend) fifths.
Methods
Ethics Statements
- Ethical approval was granted in February 2004 by the South West Wales Research Ethics Committee on behalf of NHS Wales (project ref. 2004.024). Women were given written information on the trial and data collection, and gave informed, signed consent at 36 weeks’ gestation.
- Data held in SAIL databases are anonymised and aggregated and have been obtained with permission of relevant Data Protection Officers, as approved by the National Research Ethics Service, Wales [47], [48].
The Trial
As reported elsewhere [35], [36], this randomised, double-blind, placebo-controlled, parallel-group trial assessed the effects of probiotic food supplements on key immune parameters and prevention of atopy and atopic conditions (asthma, eczema and allergic rhinitis) in young children. Healthy women with normal singleton pregnancies under the care of clinicians in Abertawe Bro Morgannwg University Health Board, Wales, UK were recruited May 2005- October 2007. All participants were ambulatory, managed in primary care, and well or “free from disease” at recruitment, although many infants were at high or increased risk of developing atopic conditions. Inclusion criteria were: mother aged ≥16 years, normal singleton pregnancy, gestation at delivery >36 weeks, freely given, signed, informed consent to participate in the study. We excluded: women unable or unwilling to give informed consent, those with any serious medical condition affecting the woman or infant or the likely outcome of the pregnancy, families where a member of the infant’s sibship or household was already recruited to the study. Women were asked to take the probiotic supplement daily from recruitment at 36 weeks until delivery, and administer the supplement to their infants daily from birth to age 6 months.
Sample Size
A sample of 308 infants (154 in each group) was sufficient to detect a 50% reduction in eczema frequency (40% to 20%) in the probiotic group [37] with 90% power and 1% significance. To demonstrate a similar proportional reduction in asthma at 5 years (20% to 10%) [38], 538 infants would have been required. We recruited 454 pregnant women within available resources.
Recruitment Strategy
A multifaceted recruitment strategy was designed to contact the whole population of pregnant women in the catchment area (Table 1). Most (362, 79.7%) participants were recruited by one of seven fieldworkers, minimising the impact of the approach style of individual researchers. Written information indicated that the trial was focussed on prevention of eczema and asthma in infants and children, who were at either increased or normal risk of developing atopy. The risk factor considered was one or more family member already suffering from an atopic condition (asthma, eczema or allergic rhinitis).
Research Clinics
When infants reached 6 months and 2 years of age, carers were invited to research clinics. Participants were informed at recruitment and reminded at invitation that separate signed, informed consent would be sought for skin-prick testing for common allergens (housemite, grass, cow’s milk, egg, cat) and, at 6 months only, for blood sample collection from the infant for immunological investigation. Interpretations of skin-prick testing were offered to carers immediately and could be used to modify exposure to common allergens. To minimise attrition, considerable efforts were made to contact participants, and where infants were unable to attend clinics, information was obtained by home visits or telephone interviews [39].
Data Collection
Trial data were obtained from several sources:
- Questionnaires (covering demographics, compliance, risk factors for atopy, signs and symptoms of atopic conditions, adverse events [40] and infant’s health) at: 36 weeks of pregnancy (recruitment), 6, 12, 18 weeks, 6 months, 1 and 2 years. At the 6 month contact, researchers asked five questions to elicit parents’ reasons for joining the trial, and their views on the trial. Responses to open questions were recorded for illustration.
- Medical records: maternity and child health.
- Biological Samples: maternal blood at 36 weeks’ gestation, infant blood from the umbilical cord and venepuncture at 6 months, placental tissue, breast milk at 2 and 6 weeks, stool samples at birth, 2, 6, 12, 18 weeks and 6 months.
- Procedures: clinical examination and skin-prick tests for common allergens at 6 months and 2 years.
No information was available on non-respondents, so summary statistics relating to the target population were obtained for comparison [41] from all publically available sources:
- 2001 Census [42] for occupation, ethnicity, and household status. Parents’ most recent occupations were coded and grouped in accordance with Office of National Statistics (ONS) [42]–[44];
- Infant Feeding Survey [45] for smoking and alcohol use;
- Welsh Health Survey [46] for asthma;
- All-Wales health services’ electronic database (Secure Anonymised Information Linkage [SAIL] database) [47], [48] for material deprivation, as Townsend scores, ranks and fifths. Townsend scores are calculated from rates of unemployment, vehicle ownership, home ownership, and overcrowding for each geographical area of residence, using Lower Super Output Areas (LSOAs) defined by postcodes. [49]. We generated a comparator group within SAIL defined by:
- precise geographical area of residence at birth, using LSOAs.
- births during the recruitment period (May 2005 to November 2007).
Data were entered into IBM SPSS statistics v19 for Windows, in duplicate. Files were compared electronically (SPSS Data Entry Builder v4) and discrepancies reconciled before analysis.
Analysis
- a) The recruited sample was compared with external population data, listed above.
- Retention, clinic attendance, consent to skin-prick testing for allergy, retention at 6 months and 2 years, and blood sample donation were explored in bivariate analyses, and, where feasible, by logistic regression [50], with variables as listed (Tables S1, S2). Regression models were built iteratively using i) socio-demographic variables ii) variables reflecting leverage, such as rashes, and reasons for joining the trial, such as altruism. Model parameters for each stage were compared. We checked for any attrition bias linked to trial arm.
- Further analyses of the trial outcomes were undertaken to explore the potential impact of volunteer bias, as recommended [4]. Trial outcome data were weighted to reflect the distribution of deprivation (Townsend) fifths amongst respondents for each outcome relative to the target population matched in the SAIL database. The weighting factor for each fifth was calculated as that fifth’s proportion in the population divided by the proportion in the sample for each outcome (weighting factor = % in population/% in sample). SPSS statistics then created a new frequency variable by multiplying existing frequencies by the weighting factor. Associations between trial arm and clinical outcome were re-tested. Subgroups were used solely to explore the findings.
Results
Between April 2005 and June 2007, 1419 expressions of interest were received, yielding 454 recruits (32%, 454/1419). Over the 2.25 years of recruitment, this 1419 represents almost 2% of the ∼74,000 births in Wales, and 20% of the 6,893 women delivering in the LSOAs represented in the trial as identified in the SAIL database. Attrition was 5.3% (24/454) at 6 month contact (Figure 1) and 16.3% (74/454) at 2 years (Figure 2).
1) Potential for Volunteer Bias
a) Recruitment.
The recruited sample was less materially deprived than the target population closely matched for area of residence at birth (Table 2). A disproportionate number of recruits were from the least deprived (Townsend) fifth. Occupational group distributions differed between trial participants and the population of South West Wales in the 2001 Census [42] (Tables S3, S4). Both these differences intensified as the trial progressed (Figures 3, 4).
Census data [42] indicated that ethnic minorities were not under-represented. We recruited relatively few lone parents (19, 4.2%), when compared with households containing children of all ages in South West Wales (7.53%). No-one was classified as ‘homeless’ at recruitment. Comparisons with pregnant women in Wales suggest that the recruited sample may over-represent non-smokers (Figure 5) and alcohol abstainers [45] (Table 2).
Most, 69% (286/417 responding to the question) participants reported first hearing about the trial when they were approached by researchers in antenatal clinics. This personal approach was crucial to the decision to enrol for most participants (233/404 responding, 58%). The hope of preventing asthma or eczema in their infant was parents’ most frequently cited reason for joining, followed by interest in eczema, asthma or allergy (Table S2). Altruistic motives were also apparent: 47% (190/403) stated that helping research and 42% (166/398) that helping children were important motivators, as illustrated:
Anything to help prevent the children of the next generation developing allergies. (Participant 147, full participation).
But these considerations could be over-ridden:
I wanted to help find a cure for eczema, but my family said we were being used as ‘guinea pigs’, so I stopped. (participant 118, telephone follow up).
Of reasons for joining the trial considered (Table S2), only ‘interest in probiotics’ was associated with occupational group (χ2 8.55, p = 0.003, df = 1) or deprivation (Townsend) fifth (χ2 4.27, p = 0.04, df = 1).
b) Retention.
Internal comparisons indicated that the most disadvantaged were less likely to be retained at 6 months. Comparisons using ONS categories and deprivation (Townsend) fifths gave similar findings: 17/24 (70.8%) lost came from ONS category 3 (routine occupations and unemployed) compared with 149 of 449 (33.2%) recruited (χ2 14.46 df = 1, OR 0.19, 0.08–0.46, p<0.001); 13/24 (54.2%) were from the most materially deprived fifth compared with 123/454 (27.1%) recruited (χ2 8.01, df = 1, OR 0.29, 0.13–0.67, p = 0.01) (Figure 6, Table S1).
Attrition was higher amongst smokers (χ2 14.38, df = 1, OR 0.20, 0.09–0.46, p<0.001). Five of the 19 (26.3%) single mothers were lost to follow up. At 2 years, in multivariate analysis, retention at 2 years was associated with maternal age, not smoking at recruitment, occupation category, carers’ reports of rashes and reporting any adverse events during supplementation (Table 3).
Clinic Attendance and consent to skin-prick testing were necessarily closely linked. Demographic variables predicted clinic attendance and consent to testing at 6 months and 2 years (Table 3, Figures 3, 4, 5). Only the most disadvantaged categories (deprivation (Townsend) fifth and ONS category 3, routine occupations or never worked) were associated with non-attendance and declining testing. Logistic regression model parameters improved when putative motivations for clinic attendance or leverage, such as reports of rashes or adverse events, and reasons for joining the trial relating to saliency and altruism, such as ‘interest in probiotics’, were taken into consideration (Table 4). ‘Wanting to help research’, predicted involvement at 6 months, but not at 2 years (Table 3).
Consent to venous blood sample donation by infants at 6 months was positively associated with professional or managerial occupations, not smoking, being in the intervention arm, interest in the trial intervention, wanting to help children and be involved in research, and experiencing asthma in adulthood. It was negatively associated with maternal use of corticosteroids (Table 3). Including factors related to reasons for joining the trial reflecting altruism, such as ‘wanting to help children’, strengthened the regression outputs. However, infants’ rashes and adverse events were not associated with consent (Table 4).
2) Data Weighting to Assess Volunteer Bias
Weighting the data according to the distribution of material deprivation (Townsend) fifths relative to the SAIL database (Table S5) changed some trial findings, but not others (Table 5). Post hoc subgroup analyses of deprivation (Townsend) fifths indicated that findings were unchanged where the impact of the intervention was concentrated in the under-represented group, the most deprived (atopic sensitisation). However, findings were modified where impact of the intervention was concentrated in the over-represented group, the least deprived (atopic eczema). The absolute risk reduction was changed by 36.2%, from 6.9% (0.9–13.1%) to 4.6% (−1.4–10.5%), and the odds ratio by 40%, from 0.40 (0.18–0.91) to 0.56 (0.26–1.21) (Table 5). Doctor-diagnosed eczema appeared to be more common in the least deprived participants, and asthma in the most deprived, but differences were not statistically significant. Interaction terms between treatment arm and material deprivation (as Townsend fifths) were not significant.
Discussion
Potential for volunteer bias, created at recruitment, intensified throughout the trial (Figures 3, 4, 5). Retention and participation were associated with socio-demographic variables, smoking status and variables reflecting leverage, saliency and altruism. Trial findings were modified by data weighting to account for volunteer bias (Table 5).
Limitations and Strengths
From single site research, we cannot assume that respondents and response patterns are representative of other populations. Unusually for a clinical trial [51], the lead institutions are in an area of the European Union (EU) where GDP is 75% below the community average, a Convergence area [52]. Trial location may have influenced recruitment, retention, and sample donation. For example, attitudes towards blood donation differ between communities [53]–[55].
This trial was restricted to healthy dyads. To our knowledge, predictors of carers’ consent to blood sample donation by well infants have not been explored in other trials, and associations reported here require testing in other populations [50]. However, cohort studies report similar clinic attendance rates [56]. The balance between benefit and harm is more uncertain in prevention or vaccine trials involving healthy participants than in therapeutic trials [57]. Further work is needed to explore generalisation of these findings to trials involving unwell or hospitalised children, where recruitment is restricted to closely defined populations with current medical conditions [14], [58]–[60].
Comparison with external data was the only option available to evaluate demographic representation at recruitment; however, some ages, locations and time-frames were not entirely congruent. Therefore, we tested this approach by comparing the deprivation scores and rankings of respondents with those of women giving birth in the same timeframe and geographical areas. We are unaware of other trials testing sample selection using this approach. The similarity between the comparisons indicates that it would be reasonable to assess volunteer bias using Census data where closely matched population data is unavailable.
The data sources used for comparison are themselves vulnerable to social desirability, volunteer and non-response bias, and may not be fully representative of the population. For example, the 2001 Census had a 93–94% response rate in Wales, falling below 90% for women aged 20–24 [61]: the most disadvantaged are likely to be under-represented [62]. Accordingly, our calculations may underestimate demographic imbalance. Reports of behaviour are vulnerable to social desirability response biases, but we have no reason to assume that our data would be uniquely vulnerable.
Non-contact bias should be distinguished from volunteer bias [13]. The recruited sample’s composition may have been influenced by the characteristics of women attending ante-natal clinics and community groups. Marginalised women may not access care or only accept domiciliary care in refuges, so would neither have received our invitation letters nor been approached (Table 1).
Interpretation of weighted analyses rests with readers; this strategy to account for volunteer or non-response bias is routine in observation studies, including UK birth cohorts [25], [63]–[65]. We acknowledge the limitations of post hoc subgroup analyses [66], [67], and present these solely to illustrate how outcome distribution affects data weighting, not to guide clinical practice. Low numbers in outcome variables necessitate cautious interpretation; however, these findings merit exploration in pooled data sets and meta-analysis.
1. Potential for Volunteer Bias
- Recruitment strategies in this trial favoured wealthier families with healthier behaviours, as in observation studies [17]–[19], [23]–[25], cluster [26] and adult prevention trials [10], [12], [27]. Significant degrees of sub-optimal recruitment and potential volunteer bias are relatively recent phenomena [68], [69]. Just as recruitment to trials is becoming increasingly difficult [2], successive UK birth cohorts have had lower response rates. While the 1958 & 1970 MRC cohorts recruited 98.76% & 95.86% (17416/17634 & 16571/17287) of those approached [70], [71], the Millenium Cohort had a 68% unweighted response rate (72% in Wales) [25].
- Retention was influenced by socio-demographic and less tangible factors.
- The most disadvantaged and smokers were less likely to participate in follow-up, attend clinics, consent to skin-prick testing or blood sample donation. Treatment allocation had no negative impact.
- ii) Potential for volunteer bias in the retained samples was not confined to socio-demographic parameters [72], [73]. Multivariate analyses indicated that when demographics were accounted, leverage, saliency [13], [74] and altruism [14], [75] are important predictors of participation (Table 4). To our knowledge, this has not been tested in trial data.
The saliency and leverage of the trial, clinic or skin-prick testing, and the theory of social exchange [74], [76], [77] featured in binary, threshold decisions to participate. Opportunities to see consultant paediatricians and receive allergen testing may have been particularly attractive to carers of infants experiencing adverse events or rashes. Access to treatment [22] or expectation of better attention incentivise participation [59], [78].
Altruism was important in the decision to consent to venous blood sample donation by well infants. Here, there were no possible direct benefits to the family, and the infant’s discomfort was a deterrent [79]. Leverage related to clinic attendance and skin-prick testing was discounted, and ‘wanting to help children’ predicted consent. Requests for time and biological samples deter many potential trial participants [1], [17], [30], [80], [81]. However, 220 participants consented to sample donation. Such altruism is more evident in less recent trials [75].
2. Volunteer Bias in Trials and Data Weighting
Applying the concept of volunteer bias to trial data tests the generalisability, external validity, transferability, utility and dependability of trial findings. Keyword searches in three databases (PubMed, Web of Science, Scopus) indicate that data weighting to account for and quantify potential volunteer bias is rarely undertaken in paediatric prevention trials.
Generalising the Findings
Findings (Table 5) suggest that to minimise any risk that results may be distorted by systematic differences between participants and the population likely to use the trial’s findings, outcomes should be assessed in samples as free of volunteer bias as possible [7], [26]. Although an unrepresentative sample does not necessarily mean that findings would not be replicated in a wider population, research quality criteria include non-biased sample selection [82]. This is particularly important where participants’ characteristics influence study outcomes [8], [13], [19], [32], [83]. Strategies to account for missing data, such as sensitivity analysis, do not address volunteer bias [84]. It cannot be assumed that participation and attrition are random events, prompting calls for full details of target or eligible populations to be reported for all trials [20].
Power of the Trial: Recruiting the Target Population
Problems were confined to the most materially disadvantaged and smokers. Non-targeted recruitment and retention risk volunteer bias and disenfranchisement of the least affluent and most marginalised, where childhood ill-health is concentrated [85]. Many outcomes in health services’ research, including childhood asthma and wheezing, are affected by material deprivation [16], [85], [86], or geographical location [87]. Eczema is associated with urbanisation [88] and parents’ educational attainment [89], both linked with reduced deprivation. Here, doctor-diagnosed eczema was no less common in the over-represented group (the affluent) (Table 5), indicating that volunteer bias did not reduce the study’s power for this outcome. However, asthma was less common in the over-represented group. For this outcome, it will be important to consider any potential loss of power, as the event rate proportion may differ between the population and the recruited and retained samples.
Robust Trial Findings: Suggestions and Solutions
Weighting increased the leverage of data from the most deprived participants (Table 5). Accordingly, this confirmed the robustness of positive outcomes concentrated in under-represented groups (atopic sensitisation). However, where the intervention’s impact was concentrated in over-represented groups (atopic eczema), weighting changed both the absolute and relative effects of the intervention. Weighting techniques, standard practice in cohort studies [25], [63]–[65], based on demographic distribution at recruitment, can augment analyses of trial data [8], [41]. Such weighting is based on assumptions that participants from disadvantaged groups are representative or typical of their groups in all respects, including attributes not recorded; only careful fieldwork and local knowledge can support such suppositions. Obviating any need for such subjective judgments, and obtaining trial evidence on which to base practice recommendations to the wider, target population, necessitates engagement, recruitment and retention of fully representative samples [34], [82]. Strategies include:
- Additional resources. Trialists are under pressure to recruit to safeguard their sponsors’ investments. However, the disadvantaged are disproportionately hard to reach [13], [21]. To safeguard investment in clinical trials, the research community should budget sufficient time and resources for complicated, personalised contact and follow up procedures [18], [28], as in birth cohorts [25], [63].
- Stratification of the population and over-sampling those least likely to participate, as in cohort studies [25], [63]–[65].
- Electronic follow up using routinely collected health services’ data, where available. More work is needed to evaluate this approach and assess the traceability of respondents.
- Weighted analysis to account for residual problems. Accounting for all possible confounders will be difficult, but even partial mapping strengthens the analysis [41].
Conclusions
If trial evidence is to reflect population diversity, demographically representative samples should be recruited and retained. Disproportionate socio-demographic representation arising at recruitment intensified throughout the trial. Accounting for this by data weighting to assess volunteer bias modified important trial findings. Whether this would occur in other trials warrants investigation. However, material deprivation is not the only predictor of participation. The leverage-saliency theory of research participation remains important; additionally, these findings indicate that altruism should not be discounted. Application of the concept of volunteer bias to clinical trials suggests that to offer reassurance regarding the generalisability, external validity, transferability, utility and dependability of findings, researchers should quantify differences between recruited samples and target populations and weight data to protect findings from potential distortion by volunteer bias.
Supporting Information
Table S1.
Variables entered into regression models, whole sample and sample retained at 6 months.
https://doi.org/10.1371/journal.pone.0067912.s001
(DOC)
Table S2.
Reasons for joining the trial (n = 430).
https://doi.org/10.1371/journal.pone.0067912.s002
(DOC)
Table S3.
Occupational groups in recruited sample and 2001 Census for South West Wales: mothers.
https://doi.org/10.1371/journal.pone.0067912.s003
(DOC)
Table S4.
Occupational groups in recruited sample and 2001 Census for South West Wales: fathers.
https://doi.org/10.1371/journal.pone.0067912.s004
(DOC)
Table S5.
Proportions used for Data Weighting for each outcome by Deprivation (Townsend) Quintile.
https://doi.org/10.1371/journal.pone.0067912.s005
(DOC)
Acknowledgments
This study uses anonymised data held in the Secure Anonymised Information Linkage (SAIL) system, which is part of the national e-health records research infrastructure for Wales. We should like to acknowledge all the data providers who make anonymised data available for research
Thanks are due to: our participants; colleagues in Abertawe Bro Morgannwg University Health Board, particularly Vivienne Davies, phlebotomist, Allyson James, clinic nurse; colleagues in Swansea University, particularly research assistants and data managers Sally Williams, Julia Kramer, Amanda Cook, Ceri Bradshaw, Ioan Humphries, and Claire Burrows.
Data are available from the authors on request.
Author Contributions
Conceived and designed the experiments: SJ SA SP CT GM. Performed the experiments: SJ MS SA CB IG MH RJ GM. Analyzed the data: SJ AW IR. Contributed reagents/materials/analysis tools: CB MH IG SP. Wrote the paper: SJ AW MS SP IR GM. Commented on drafts and approved the final manuscript: SJ AW MS SA CB IG MH RJ SP IR CT GM.
References
- 1.
Institute of Medicine (US) (2012) Public engagement and clinical trials: New models and disruptive technologies: Workshop summary. Washington (DC): National Academies Press (US). Available: http://www.ncbi.nlm.nih.gov/books/NBK91498/pdf/TOC.pdf. Accessed 2012 Aug 14.
- 2. Eisner J, Jones L (2009) Patient recruitment: Are we looking in the right place? Clinical Discovery 4(2): 19–21.
- 3. Wahlbeck K, Tuunainen A, Ahokas A, Leucht S (2001) Dropout rates in randomised antipsychotic drug trials. Psychopharmacology (Berl) 155(3): 230–233.
- 4. O’Neill RT, Temple R (2012) The prevention and treatment of missing data in clinical trials: An FDA perspective on the importance of dealing with it. Clin Pharmacol Ther 91(3): 550–4
- 5.
Higgins JPT, Altman DG, Sterne JAC, editors (2011) Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available: www.cochrane-handbook.org. Accessed 2012 Dec 15.
- 6.
PubMed MeSH database (1990) Selection bias. Available: http://www.ncbi.nlm.nih.gov/mesh?term=selection%20bias. Accessed 2012 Dec 14.
- 7.
Bartlett C, Doyal L, Ebrahim S, Davey P, Bachmann M, et al.. (2005) The causes and effects of socio-demographic exclusions from clinical trials. Health Technol Assess 9(38): iii-iv, ix-x, 1–152.
- 8. Dekkers OM, von Elm E, Algra A, Romijn JA, Vandenbroucke JP (2010) How to assess the external validity of therapeutic trials: A conceptual approach. Int J Epidemiol 39(1): 89–94.
- 9. Frangakis C (2009) The calibration of treatment effects from clinical trials to target populations. Clin Trials 6(2): 136–140.
- 10. Martinson BC, Crain AL, Sherwood NE, Hayes MG, Pronk NP, et al. (2010) Population reach and recruitment bias in a maintenance RCT in physically active older adults. J Phys Act Health 7(1): 127–35.
- 11. Sackett DL (1979) Bias in analytic research. J Chronic Dis 32(1–2): 51–63.
- 12. Remington RD, Taylor HL, Buskirk ER (1978) A method for assessing volunteer bias and its application to a cardiovascular disease prevention progamme involving physical activity. J Epidemiol Community Health 32(4): 250–255.
- 13. Groves RM (2006) Non-response rates and non-response bias in household surveys. Public Opinion Quarterly 70(5): 646–675.
- 14. Harth SC, Thong YH (1990) Sociodemographic and motivational characteristics of parents who volunteer their children for clinical research: A controlled study. BMJ 300(6736): 1372–1375.
- 15. Keeter S, Miller C, Kohut A, Groves RM, Presser S (2000) Consequences of reducing nonresponse in a national telephone survey. Public Opin Q 64: 125–148.
- 16. Kho ME, Duffett M, Willison DJ, Cook DJ, Brouwers MC (2009) Written informed consent and selection bias in observational studies using medical records: Systematic review. BMJ 338: b866.
- 17. Mihrshahi S, Vukasin N, Forbes S, Wainwright C, Krause W, et al. (2002) Are you busy for the next 5 years? Recruitment in the childhood asthma prevention study (CAPS). Respirology 7(2): 147–151.
- 18. Park ER, Quinn VP, Chang Y, Regan S, Loudin B, et al. (2007) Recruiting pregnant smokers into a clinical trial: Using a network-model managed care organization versus community-based practices. Prev Med 44(3): 223–229.
- 19. DiMartino LD, Hammill BG, Curtis LH, Gottdiener JS, Manolio TA, et al. (2009) External validity of the cardiovascular health study: A comparison with the Medicare population. Med Care 47(8): 916–923.
- 20. Toerien M, Brookes ST, Metcalfe C, de Salis I, Tomlin Z, et al. (2009) A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials 10: 52
- 21. Shilling V, Williamson P, Hickey H, Sowden E, Smyth R, et al. (2011) Processes in recruitment to randomised controlled trials of medicines for children (RECRUIT): A qualitative study. Health Technol Assess 15(15): 1–116.
- 22. Rothmier JD, Lasley MV, Shapiro GG (2003) Factors influencing parental consent in pediatric clinical research. Pediatrics 111: 1037–1041.
- 23. Kotaniemi JT, Hassi J, Kataja M, Jonsson E, Laitinen LA, et al. (2001) Does non-responder bias have a significant effect on the results in a postal questionnaire study? Eur J Epidemiol 17(9): 809–817.
- 24.
University of Bristol (2008) ALSPAC Avon longitudinal study of parents and children. Reponse rates and biases. Available: http://www.bristol.ac.uk/alspac/sci-com/resource/represent/. Accessed 2012 Aug 2.
- 25.
Dex S, Joshi H, editors (2004) Millennium cohort study first survey: A user’s guide to initial findings. Centre for Longitudinal Studies, Institute of Education, University of London. Available: http://eprints.ioe.ac.uk/5933/1/MCS1_A_Users_Guide_To_Initial_Findings.pdf. Accessed 2012 Aug 2.
- 26. Vuchinich S, Flay BR, Aber L, Bickman L (2012) Person mobility in the design and analysis of cluster-randomized cohort prevention trials. Prev Sci 13(3): 300–313.
- 27. Daniels LA, Wilson JL, Mallan KM, Mihrshahi S, Perry R, et al. (2012) Recruiting and engaging new mothers in nutrition research studies: lessons from the Australian NOURISH randomised controlled trial. Int J Behav Nutr Phys Act 9(1): 129
- 28. Jordan S, Morgan G (2011) Recruitment to paediatric trials. The Welsh Paediatric Journal 35: 36–40.
- 29. Britton A, McKee M, Black N, McPherson K, Sanderson C, et al. (1999) Threats to applicability of randomised trials: Exclusions and selective participation. J Health Serv Res Policy 4(2): 112–121.
- 30. Juni P, Altman DG, Egger M (2001) Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 323: 42–46.
- 31.
Prinz RJ, Smith EP, Dumas JE, Laughlin JE, White DW, et al.. (2001) Recruitment and retention of participants in prevention trials involving family-based interventions. Am J Prev Med 20(1 Suppl): 31–7.
- 32. Fewtrell MS, Kennedy K, Singhai A, Martin RM, Ness A, et al. (2008) How much loss is acceptable in long-term randomised trials and prospective studies? Arch Dis Child 93(6): 458–461.
- 33. Dhruva SS, Redberg RF (2008) Variations between clinical trial participants and Medicare beneficiaries in evidence used for Medicare national coverage decisions. Arch Intern Med 168(2): 136–140.
- 34.
Department of Health (2005) Research governance framework for health and social care. 2nd edition. Available: http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_4122427.pdf. Accessed 2012 Aug 4.
- 35. Allen SJ, Jordan S, Storey M, Thornton CA, Gravenor M, et al. (2010) Dietary supplementation with Lactobacilli and Bifidobacteria is well tolerated and not associated with adverse events during late pregnancy and early infancy. J Nutr 140(3): 483–488.
- 36.
Allen SJ, Jordan S, Storey M, Thornton C, Gravenor M, et al.. (2012) Probiotics and atopic eczema: A double-blind randomised controlled trial. Arch Dis Child 97 S1 A2 P05. doi:10.1136/archdischild-2012-301885.5.
- 37. Kalliomäki M, Salminen S, Arvilommi H, Kero P, Koskinen P, et al. (2001) Probiotics in the primary prevention of atopic disease: A randomised placebo-controlled trial. Lancet 357: 1076–1079.
- 38. Kaur B, Anderson HR, Austin J, Burr M, Harkins LS, et al. (1998) Prevalence of asthma symptoms, diagnosis, and treatment in 12–14 year old children across Great Britain (international study of asthma and allergies in childhood, ISAAC UK). BMJ 316: 118–124.
- 39. Akl EA, Briel M, You JJ, Sun X, Johnston BC, et al. (2012) Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): Systematic review. BMJ 344: e2809
- 40.
International Conference on Harmonisation (ICH) 1996 ICH harmonised tripartite guideline for good clinical practice E6(R1). Marlow: Institute of Clinical Research. Available: http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6_R1/Step4/E6_R1__Guideline.pdf. Accessed 2012 Aug 3.
- 41. Cole SR, Stuart EA (2010) Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. Am J Epidemiol 172(1): 107–115.
- 42.
Office for National Statistics (ONS) (2003) Census 2001: Key statistics for Assembly constituencies and Assembly electoral regions for the National Assembly for Wales. Newport: Office for National Statistics. Available: http://www.ons.gov.uk/ons/search/index.html?pageSize=50&sortBy=none&sortDirection=none&newquery=Wales2001censusstatisticsforassemblyconstituenciesandassemblyelectoralregions. Accessed 2012 Aug 4.
- 43.
Office for National Statistics (ONS) (2005) The national statistics socio-economic classification user manual. Basingstoke: Palgrave Macmillan. Available: http://www.ons.gov.uk/ons/guide-method/classifications/archived-standard-classifications/soc-and-sec-archive/the-national-statistics-socio-economic-classification-user-manual.pdf. Accessed 2012 Dec 15.
- 44.
Office for National Statistics (ONS) (2005) Standard occupational classification and socio-economic classification: Archive. Derivation tables based on standard occupational classification 2000. Available: http://www.ons.gov.uk/ons/guide-method/classifications/archived-standard-classifications/soc-and-sec-archive/index.html. Accessed 2012 Dec 15.
- 45.
Bolling K, Grant C, Hamlyn B, Thornton A (2007) Infant Feeding Survey 2005. London: Government Statistical Services, Department of Health.
- 46.
Dolman R, Roberts C, Kingdon A, editors (2008) Welsh health survey 2007. Cardiff: Welsh Assembly Government. Available: http://wales.gov.uk/topics/statistics/publications/publication-archive/healthsurvey2007/?lang=en. Accessed 2012 Aug 2.
- 47. Lyons RA, Jones KH, John G, Brooks CJ, Verplancke JP, et al. (2009) The SAIL databank: Linking multiple health and social care datasets. BMC Med Inform Decis Mak 9: 3
- 48. Ford DV, Jones KH, Verplancke JP, Lyons RA, John G, et al. (2009) The SAIL databank: building a national architecture for e-health research and evaluation. BMC Health Serv Res 9: 157
- 49.
Townsend P, Phillimore P, Beattie A (1988) Health and deprivation: Inequality and the North. London: Routledge.
- 50. Royston P, Moons KG, Altman DG, Vergouwe Y (2009) Prognosis and prognostic research. Developing a prognostic model. BMJ 338: b604.
- 51. Jordan S, Segrott J (2008) Evidence-based practice: The debate. J Nurs Manag 16(4): 385–387.
- 52.
European Commission (2008) Objective 1: Supporting development in less prosperous regions. Available: http://ec.europa.eu/regional_policy/archive/objective1/index_en.htm. Accessed 2012 Aug 4.
- 53. Godin G, Sheeran P, Conner M, Germain M, Blondeau D, et al. (2005) Factors explaining the intention to give blood among the general population. Vox Sang 89(3): 140–149.
- 54. McMahon R, Byrne M (2008) Predicting donation among an Irish sample of donors and nondonors: Extending the theory of planned behavior. Transfusion 48(2): 321–331.
- 55. Duke NN, Skay CL, Pettingell SL, Borowsky IW (2009) From adolescent connections to social capital: Predictors of civic engagement in young adulthood. J Adolesc Health 44(2): 161–168.
- 56. Perkin MR, Strachan DP, Hc W, Lack G, Golding J (2006) ALSPAC Study Team. The predictive value of early life total immunoglobulin E measurement in identifying atopic children in a population-based birth cohort study. Pediatr Allergy Immunol 17(2): 118–124.
- 57. Chantler TE, Lees A, Moxon ER, Mant D, Pollard AJ (2007) The role familiarity with science and medicine plays in parents’ decision making about enrolling a child in vaccine research. Qual Health Res 17: 311–322.
- 58. Hoehn KS, Wernovsky G, Rychik J, Gaynor JW, Spray TL, et al. (2005) What factors are important to parents making decisions about neonatal research? Arch Dis Child Fetal Neonatal Ed 90: F267–269.
- 59. Fisher HR, McKevitt C, Boaz A (2011) Why do parents enroll their children in research: A narrative synthesis. J Med Ethics 37(9): 544–551.
- 60. Shilling V, Young B (2009) How do parents experience being asked to enter a child in a randomised controlled trial? BMC Med Ethics 16: 10–11.
- 61.
ONS (Office of National Statistics) (2003) Data and Products: Quality of the Census: Response rates. Newport: ONS. Available: http://www.ons.gov.uk/ons/guide-method/census/census-2001/data-and-products/quality-of-the-census-data/response-rates/index.html. Accessed 2012 Dec 14.
- 62. Majeed FA, Cook D, Poloniecki J, Martin D (1995) Using data from the 1991 census. BMJ 310: 1511–1514.
- 63.
Hansen K, Joshi H, editors (2008) Millennium Cohort Study third survey: A user’s guide to initial findings. London: Centre for Longitudinal Studies (CLS) Institute of Education, University of London. Available: http://eprints.ioe.ac.uk/5931/1/MCS_3_Descriptive_Report_Oct_2008.pdf. Accessed 2012 Aug 2.
- 64. Schmoor C, Gall C, Stampf S, Graf E (2011) Correction of confounding bias in non-randomized studies by appropriate weighting. Biom J 53(2): 369–387
- 65. Seaman SR, White IR, Copas AJ, Li L (2012) Combining multiple imputation and inverse-probability weighting. Biometrics 68(1): 129–137
- 66. Lagakos SW (2006) The Challenge of subgroup analysis – reporting without distorting. N Engl J Med 354 (16): 1667–1669.
- 67. Sun X, Briel M, Busse JW, You JJ, Akl EA, et al. (2012) Credibility of claims of subgroup effects in randomised controlled trials: Systematic review. BMJ 344: e1553
- 68. McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, et al. (2006) What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials 7: 9.
- 69. Elwood PC, Haley TJ, Hughes SJ, Sweetnam PM, Gray OP, et al. (1981) Child growth (0–5 years), and the effect of entitlement to a milk supplement. Arch Dis Child 56(11): 831–835.
- 70. Power C, Elliott J (2006) Cohort profile: 1958 British birth cohort (National Child Development Study). Int J Epidemiol 35(1): 34–41.
- 71. Elliott J, Shepherd P (2006) Cohort profile: 1970 British Birth Cohort (BCS70). Int J Epidemiol 35: 836–843.
- 72.
Campbell MK, Snowdon C, Francis D, Elbourne D, McDonald AM, et al.. (2007) Recruitment to randomised trials: Strategies for trial enrolment and participation study. The STEPS study. Health Technol Assess 11(48): iii, ix-105.
- 73.
Robinson EJ, Kerr CE, Stevens AJ, Lilford RJ, Braunholtz DA, et al.. (2005) Lay public understands of equipoise and randomisation in randomised controlled trials. Health Technol Assess 9(8): 1–192, iii–iv.
- 74. Groves RM, Singer E, Corning A (2000) Leverage-saliency theory of survey participation: Description and an illustration. Public Opin Q 64(3): 299–308.
- 75. Tooher RL, Middleton PF, Crowther CA (2008) A thematic analysis of factors influencing recruitment to maternal and perinatal trials. BMC Pregnancy Childbirth 8: 36.
- 76. McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, et al. (2001) Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 5(31): 1–256.
- 77. Krosnick J (1999) Survey research. Annu Rev Psychol 50: 537–567.
- 78. Olson L (2002) Patient-centred equipoise and the ethics of randomised controlled trials. Monash Bioethics Review 21(2): 55–67.
- 79. Snowdon C, Elbourne D, Garcia J (2006) “It was a snap decision”: parental and professional perspectives on the speed of decisions about participation in perinatal randomised controlled trials. Soc Sci Med 62(9): 2279–2290.
- 80. Fitzgibbon ML, Prewitt TE, Blackman LR, Simon P, Luke A, et al. (1998) Quantitative assessment of recruitment efforts for prevention trials in two diverse black populations. Prev Med 27(6): 838–845.
- 81. Prescott RJ, Counsell CE, Gillespie WJ, Grant AM, Russell IT, et al. (1999) Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assess 3(20): 1–143.
- 82. Chou R, Aronson N, Atkins D, Ismaila AS, Santaguida P, et al. (2010) AHRQ series paper 4: Assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol 63(5): 502–512.
- 83. Zarin D, Young J, West J (2005) Challenges to evidence-based medicine: A comparison of patients and treatments in randomized controlled trials with patients and treatments in a practice research network. Soc Psychiatry Psychiatr Epidemiol 40: 27–35.
- 84. White IR, Horton NJ, Carpenter J, Pocock SJ (2011) Strategy for intention to treat analysis in randomised trials with missing outcome data. BMJ 342: d40
- 85. Propper C, Rigg J, Burgess S (2007) Child health: Evidence on the roles of family income and maternal mental health from a UK birth cohort. Health Econ 16(11): 1245–1269.
- 86. Violato M, Petrou S, Gray R (2009) The relationship between household income and childhood respiratory health in the United Kingdom. Soc Sci Med 69(6): 955–963.
- 87. McNally NJ, Williams HC, Phillips DR, Strachan DP (2000) Is there a geographical variation in eczema prevalence in the UK? Evidence from the 1958 British birth cohort study. Br J Dermatol 142(4): 712–720.
- 88. Schram ME, Tedja AM, Spijker R, Bos JD, Williams HC, et al. (2010) Is there a rural/urban gradient in the prevalence of eczema? A systematic review. Br J Dermatol 162(5): 964–973.
- 89. Shaw TE, Currie GP, Koudelka CW, Simpson EL (2011) Eczema prevalence in the United States: Data from the 2003 national survey of children’s health. J Invest Dermatol 131(1): 67–73.
- 90. Kenyon S, Dixon-Woods M, Jackson CJ, Windridge K, Pitchforth E (2006) Participating in a trial in a critical situation: A qualitative study in pregnancy. Qual Saf Health Care 15(2): 98–101.
- 91. Schoetzau A, Gehring U, Franke K, Grubl A, Koletzko S, et al. (2002) Maternal compliance with nutritional recommendations in an allergy preventive programme. Arch Dis Child 86(3): 180–184.
- 92. Nystuen P, Hagen KB (2004) Telephone reminders are effective in recruiting nonresponding patients to randomized controlled trials. J Clin Epidemiol 57(8): 773–776.
- 93. UyBico SJ, Pavel S, Gross CP (2007) Recruiting vulnerable populations into research: A systematic review of recruitment interventions. J Gen Intern Med 22(6): 852–863.
- 94. Southern DA, Lewis S, Maxwell CJ, Dunn JR, Noseworthy TW, et al. (2008) Sampling ‘hard-to-reach’ populations in health research: Yield from a study targeting Americans living in Canada. BMC Med Res Methodol 8: 57
- 95.
Mapstone J, Elbourne D, Roberts IG (2007) Strategies to improve recruitment to research studies. Cochrane Database of Systematic Reviews Issue 2. Art. No: MR000013. doi:10.1002/14651858.MR000013.pub3.
- 96. Watson JM, Torgerson DJ (2006) Increasing recruitment to randomised trials: A review of randomised controlled trials. BMC Med Res Methodol 6: 34.
- 97. Caldwell PH, Hamilton S, Tan A, Craig JC (2010) Strategies for increasing recruitment to randomised controlled trials: systematic review. PLoS Med 7(11): e1000368
- 98.
National Patient Safety Agency (NPSA) (2007) Information sheets and consent forms: Guidance for researchers and reviewers. London: NHS National Research Ethics Service.
- 99.
Medical Research Council (2004) Clinical research leaflet. London: MRC.
- 100. Wade J, Donovan JL, Athene Lane J, Neal DE, Hamdy FC (2009) Its not just what you say, its also how you say it: Opening the ‘black box’ of informed consent appointments in randomised controlled trials. Soc Sci Med 68: 2018–2028.
- 101.
Dillman D (2007) Mail and internet surveys: The tailored design method. New York: Wiley.
- 102. Francis D, Roberts I, Elbourne DR, Shakur H, Knight RC, et al. (2007) Marketing and clinical trials: A case study. Trials 20: 8–37.
- 103. Fouad MN, Corbie-Smith G, Curb D, Howard BV, Mouton C, et al. (2004) Special populations recruitment for the Women’s Health Initiative: Successes and limitations. Control Clin Trials 25(4): 335–352.