An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial

Ann Cochrane; Charlie Welch; Caroline Fairhurst; Sarah Cockayne; David J. Torgerson; OTIS Study Group

doi:10.12688/f1000research.22361.1

Home Browse An evaluation of a personalised text message reminder compared to...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Clinical trial

An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial

[version 1; peer review: 2 approved]

Ann Cochrane ¹, Charlie Welch¹, Caroline Fairhurst¹, Sarah Cockayne¹, David J. Torgerson¹, OTIS Study Group

Ann Cochrane ¹, Charlie Welch¹, [...] Caroline Fairhurst¹, Sarah Cockayne¹, David J. Torgerson¹, OTIS Study Group

PUBLISHED 26 Feb 2020

Author details Author details

¹ York Trials Unit, University of York, York, UK

Ann Cochrane
Roles: Investigation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Charlie Welch
Roles: Data Curation, Formal Analysis, Software, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Caroline Fairhurst
Roles: Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Software, Validation, Visualization, Writing – Review & Editing

Sarah Cockayne
Roles: Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Validation, Visualization, Writing – Review & Editing

David J. Torgerson
Roles: Conceptualization, Funding Acquisition, Methodology, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Studies Within A Trial (SWAT) collection.

Abstract

Background: Research outcome data is commonly collected using postal questionnaires; however, poor response can introduce bias and reduce statistical power. Text messaging is simple, cost-effective, and can be customised to the individual. Personalised, reminder text messages may improve response rates.
Methods: A two-arm, parallel group ‘Study within a Trial’ (SWAT) was embedded within the Occupational Therapist Intervention Study (OTIS), a randomised controlled trial of a home assessment for falls prevention in older people. OTIS participants who provided a mobile phone number were randomly allocated (1:1) to receive either a personalised text message (Title, Surname, plus York Trials Unit (YTU) text) or the standard YTU text alone, prior to receiving their four-month post-randomisation follow-up postal questionnaire. The primary outcome measure was the proportion of participants who returned the questionnaire. Secondary outcomes were: time to response, completeness of response, requirement of a reminder letter, and cost-effectiveness. Binary data were compared using logistic regression and time to response by Cox proportional hazards regression.
Results: A total of 403 participants were randomised: 201 to the personalised text and 202 to the standard text. Of the 283 participants included in the final analysis, 278 (98.2%) returned their questionnaire; 136 (97.8%) for the personalised text versus 142 (98.6%) for the standard text (adjusted odds ratio 0.64, 95% CI 0.10 to 3.88, p=0.63). The median time to response was nine days in both groups. In total, 271 (97.5%) participants returned a complete questionnaire; 133 (97.8%) in the personalised text versus 138 (97.2%) for the standard text. In total, 21 reminder letters were sent. The additional cost of personalised text messages was £0.04 per participant retained.
Conclusions: Personalised texts were not superior to standard texts in any outcome assessed in our study. Further SWATs are needed to perform a meta-analysis and obtain more evidence.
Registration: ISRCTN22202133; SWAT 35.

Keywords

SWAT, Randomised Controlled Trial, personalised, SMS text, postal questionnaire, reminder

Corresponding author: Ann Cochrane

Competing interests: No competing interests were disclosed.

Grant information: This study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number 14/49/149). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2020 Cochrane A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Cochrane A, Welch C, Fairhurst C et al. An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial [version 1; peer review: 2 approved]. F1000Research 2020, 9:154 (https://doi.org/10.12688/f1000research.22361.1) First published: 26 Feb 2020, 9:154 (https://doi.org/10.12688/f1000research.22361.1) Latest published: 26 Feb 2020, 9:154 (https://doi.org/10.12688/f1000research.22361.1)

Introduction

Evaluating strategies to improve the efficiency of conducting trials is a priority. Achieving high response rates for postal follow-up questionnaires is challenging; non-response threatens study validity through bias and reduced effective sample size¹. Rigorous evaluation can be achieved by undertaking a Study within a Trial (SWAT)^2–4. A SWAT is a self-contained study embedded within a host trial, which aims to evaluate an intervention⁵.

There are many strategies towards improving response to postal questionnaires including short messaging service (SMS) text prompts; however, uncertainty remains^6,7 as to their effectiveness^8–13. Furthermore, some evidence exists¹⁴ to suggest that personalised texts, in which recipients were addressed by name, increased average payment of delinquent fines compared to non-personalised texts.

Here, we report the results of a SWAT evaluating a personalised text compared to a standard (non-personalised) text on postal questionnaire response rates in an elderly population.

Methods

Design

This two-arm, parallel-group, individually randomised controlled trial (RCT) was embedded within OTIS, a UK-based modified cohort RCT of occupational therapist-led home environmental assessment for the prevention of falls in older people¹⁵. This SWAT was registered as part of the host trial (OTIS) registration (ISRCTN22202133; date registered: 20.06.2016) and with the Northern Ireland Hub for Trials Methodology Research SWAT Repository (SWAT 35; date registered: 20.02.2016).

Participants

Eligible OTIS participants who agreed to receive text communication during participation, provided a mobile number, and were due to receive their four-month post-randomisation postal questionnaire, were randomised into this SWAT.

Intervention

Participants received a single text four days after their four-month questionnaire was posted (Table 1).

Table 1. Text message content by allocation.

Embedded trial allocation	Text message sent to participants
Personalised text	“OTIS trial: [Title, Surname of participant] you should have received a questionnaire in the post by now. Your answers are important; so please help by returning it as soon as you can. Thanks.”
Standard text	“OTIS trial: You should have received a questionnaire in the post by now. Your answers are important; so please help by returning it as soon as you can. Thanks.”

Outcomes

The primary outcome was the proportion of participants who returned their four-month postal questionnaire. Secondary outcomes were: time to response, completeness of response, use of a reminder letter, and cost-effectiveness (Table 2).

Table 2. SWAT primary and secondary outcomes.

Outcome	Definition	Type
Proportion of questionnaires returned	Proportion of questionnaires returned to York Trials Unit at four months post-randomisation.	Binary (returned/not returned)
Time to questionnaire return	Number of days elapsed between the date the questionnaire was sent to participants and the date the questionnaire was recorded as being returned to York Trials Unit. Truncated at 120 days.	Time to event (0 – 120 days)
Completeness of response	Proportion of participants returning a sufficiently complete questionnaire. A returned four month questionnaire was defined as sufficiently complete if the participant provided responses to; 1) whether they had fallen in the previous four months; 2) the extent to which they had been worried about falling; 3) all five dimensions of the EQ-5D-5L.	Binary (complete/incomplete)
Reminder letters sent	Proportion of participants sent a reminder letter (and additional blank copy of the questionnaire) due to not having returned the questionnaire within 21 days.	Binary (sent/not sent)
Cost of retaining participants at four months	Total cost per participant of texts and additional contacts.	Continuous

Sample size

As is usual for embedded trials, no formal power calculation was undertaken^3–5 as the sample size was constrained by the number of participants available in the host trial.

Randomisation

Eligible participants (n=403) were randomised (1:1) using randomly varying blocks of four and six, stratified by OTIS trial group allocation. Allocations were generated by the OTIS trial statistician using Stata version 13.0, before being shared with the YTU data management staff responsible for the setup of the text messaging system. Eligible participants were then matched against the generated sequence in the order that they were randomised to the main trial.

Blinding

Participants were not aware of their involvement within this SWAT; only to the OTIS trial group allocation. Study team members performing administrative, statistical or health economic roles were also not blinded, but data entry staff were.

Ethical approval

Approvals were granted by NHS West of Scotland Research Ethics Committee 3 (ref. 16/WS/0154); the University of York, Department of Health Sciences Research Governance Committee and the Health Research Authority. Consent for the SWAT was waived by the above-named Research Ethics Committee.

Statistical analysis

Analyses were conducted in Stata version 15.0¹⁶. Baseline characteristics are summarised descriptively (Table 3). Binary outcomes were analysed using logistic regression, and time to questionnaire return was analysed using Cox proportional hazards regression. Time to return was truncated at 120 days allowing for the next follow-up time point (eight months post-randomisation) and illustrated using a Kaplan-Meier curve. Models were adjusted for SWAT and OTIS trial allocation. Unadjusted analyses of both binary and time to event outcomes are also presented. The costs incurred retaining participants are summarised descriptively (Table 5).

Table 3. Baseline characteristics of the participants included in the analysis.

Baseline characteristic	Personalised texts (N = 139)	Standard texts (N = 144)	Total (N = 283)
OTIS trial allocation, n (%)
Usual care	96 (69.1)	99 (68.8)	195 (68.9)
Intervention	43 (30.9)	45 (31.3)	88 (31.1)
Missing	0 (0.0)	0 (0.0)	0 (0.0)
Age (years)
N	139	144	283
Mean (SD)	77.8 (6.1)	76.7 (5.7)	77.3 (5.9)
Median (1^st Q, 3^rd Q)	76.8 (72.8, 81.4)	75.5 (72.3, 80.5)	76.0 (72.7, 81.1)
Sex, n (%)
Male	45 (32.4)	57 (39.6)	102 (36.0)
Female	94 (67.6)	87 (60.4)	181 (64.0)
Missing	0 (0.0)	0 (0.0)	0 (0.0)
Taking >4 prescribed medications, n (%)
Yes	61 (43.9)	69 (47.9)	130 (45.9)
No	77 (55.4)	74 (51.4)	151 (53.4)
Missing	1 (0.7)	1 (0.7)	2 (0.7)
EQ-5D-5L – Mobility, n (%)
No problems walking	49 (35.3)	67 (46.5)	116 (41.0)
Slight problems walking	37 (26.6)	27 (18.8)	64 (22.6)
Moderate problems walking	38 (27.3)	37 (25.7)	75 (26.5)
Severe problems walking	11 (7.9)	12 (8.3)	23 (8.1)
Unable to walk	0 (0.0)	1 (0.7)	1 (0.4)
Missing	4 (2.9)	0 (0.0)	4 (1.4)
EQ-5D-5L – Self-care, n (%)
No problems washing/dressing	104 (74.8)	117 (81.3)	221 (78.1)
Slight problems washing/dressing	25 (18.0)	18 (12.5)	43 (15.2)
Moderate problems washing/dressing	8 (5.8)	7 (4.9)	15 (5.3)
Severe problems washing/dressing	1 (0.7)	1 (0.7)	2 (0.7)
Unable to wash/dress myself	0 (0.0)	0 (0.0)	0 (0.0)
Missing	1 (0.7)	1 (0.7)	2 (0.7)
EQ-5D-5L – Usual activities, n (%)
No problems doing usual activities	52 (37.4)	69 (47.9)	121 (42.8)
Slight problems doing usual activities	45 (32.4)	40 (27.8)	85 (30.0)
Moderate problems doing usual activities	25 (18.0)	29 (20.1)	54 (19.1)
Severe problems doing usual activities	15 (10.8)	4 (2.8)	19 (6.7)
Unable to do usual activities	1 (0.7)	2 (1.4)	3 (1.1)
Missing	1 (0.7)	0 (0.0)	1 (0.4)
EQ-5D-5L – Pain/discomfort, n (%)
No pain or discomfort	24 (17.3)	28 (19.4)	52 (18.4)
Slight pain or discomfort	55 (39.6)	60 (41.7)	115 (40.6)
Moderate pain or discomfort	43 (30.9)	44 (30.6)	87 (30.7)
Severe pain or discomfort	14 (10.1)	11 (7.6)	25 (8.8)
Extreme pain or discomfort	0 (0.0)	1 (0.7)	1 (0.4)
Missing	3 (2.2)	0 (0.0)	3 (1.1)
EQ-5D-5L – Anxiety/depression, n (%)
Not anxious or depressed	78 (56.1)	91 (63.2)	169 (59.7)
Slightly anxious or depressed	37 (26.6)	39 (27.1)	76 (26.9)
Moderately anxious or depressed	15 (10.8)	8 (5.6)	23 (8.1)
Severely anxious or depressed	1 (0.7)	0 (0.0)	1 (0.4)
Extremely anxious or depressed	1 (0.7)	0 (0.0)	1 (0.4)
Missing	7 (5.0)	6 (4.2)	13 (4.6)
EQ-5D-5L – General health (0 – 100)*
N	139	143	282
Mean (SD)	74.6 (15.6)	75.2 (17.0)	74.9 (16.3)
Median (1^st Q, 3^rd Q)	80.0 (65.0, 85.0)	80.0 (66.0, 90.0)	80.0 (66.0, 88.0)

*0-worst health you can imagine, 100-best health you can imagine

Results

Delays setting-up the text messaging system meant no texts were sent prior to 7^th December 2017. In total 120 (29.8%) randomised participants were due texts before this date. These participants are therefore excluded from the analysis. Participants (n=283) due texts on or after this date were analysed as randomised (Figure 1).

Figure 1. The flow of participants through the embedded trial.

Results are presented in Table 4. A total of 136 (97.8%) participants in the personalised text group returned their four-month questionnaire, compared with 142 (98.6%) in the standardised text group (adjusted odds ratio (OR) 0.64, 95% CI 0.10 to 3.88, p=0.63). In total, 10 personalised text participants were sent a reminder letter and 11 in the standard text arm. Of 278 returned questionnaires, 271 (97.5%) were completed: 97.8% in the personalised arm and 97.2% in the standard text arm (adjusted OR 1.29, 95% CI 0.28 to 5.89, p=0.75).

Table 4. Analysis of binary outcomes.

Outcome	Personalised	Standard	Analysis	OR (95% CI) (personalised/standard)	p-value
Questionnaire returns	136/139 (97.8%)	142/144 (98.6%)	Unadjusted	0.64 (0.11 to 3.88)	0.63
Questionnaire returns	136/139 (97.8%)	142/144 (98.6%)	Adjusted*	0.64 (0.10 to 3.88)	0.63
Reminder letters sent	10/139 (7.2%)	11/144 (7.6%)	Unadjusted	0.94 (0.38 to 2.28)	0.89
Reminder letters sent	10/139 (7.2%)	11/144 (7.6%)	Adjusted	0.94 (0.38 to 2.28)	0.89
Complete questionnaires (returned only)	133/136 (97.8%)	138/142 (97.2%)	Unadjusted	1.29 (0.28 to 5.85)	0.75
Complete questionnaires (returned only)	133/136 (97.8%)	138/142 (97.2%)	Adjusted	1.29 (0.28 to 5.89)	0.75
Complete questionnaires (all)	133/139 (95.7%)	138/144 (95.8%)	Unadjusted	0.96 (0.30 to 3.06)	0.95
Complete questionnaires (all)	133/139 (95.7%)	138/144 (95.8%)	Adjusted	0.96 (0.30 to 3.07)	0.95

* Primary

Table 5. Costs per participant of retention at four months, by allocation and overall.

Cost	Personalised texts (N = 139)	Standard texts (N = 144)	Total (N = 283)
Cost of texts (pence)
Mean (SD)	9.5 (0.8)	4.7 (0.7)	7.1 (2.5)
Median (1^st Q, 3^rd Q)	9.6 (9.6, 9.6)	4.8 (4.8, 4.8)	4.8 (4.8, 9.6)
Min, Max	0.0, 9.6	0.0, 4.8	0.0, 9.6
Cost of reminder letters (pence)
Mean (SD)	16.9 (60.9)	18.0 (62.6)	17.4 (61.7)
Median (1^st Q, 3^rd Q)	0.0 (0.0, 0.0)	0.0 (0.0, 0.0)	0.0 (0.0, 0.0)
Min, Max	0.0, 235.0	0.0, 235.0	0.0, 235.0
Total costs (pence)
Mean (SD)	26.4 (61.0)	22.7 (62.7)	24.5 (61.8)
Median (1^st Q, 3^rd Q)	9.6 (9.6, 9.6)	4.8 (4.8, 4.8)	9.6 (4.8, 9.6)
Min, Max	0.0, 244.6	0.0, 239.8	0.0, 244.6

The median time to return was nine days in both groups. A log-rank test gave a p-value of 0.57; hence, the data provide little evidence to reject the hypothesis that the two groups have the same survival function. The Cox proportional hazards model corroborated this (hazard ratio 1.06, 95% CI 0.84 to 1.35, p=0.60) (Figure 2). Examination of the log-log plots of the estimated survival functions, and a global test of the Schoenfeld residuals suggested the proportional hazards assumption was reasonable (p=0.52).

Figure 2. Kaplan-Meier curve for time to questionnaire return.

Cost-effectiveness

Standard texts were 159 characters (costing £0.048), whereas personalised texts ranged from 166 to 178 characters (costing £0.096). Other costs included reminder letters and additional questionnaires posted to non-responders (£2.35 each) (Table 5).

Discussion

These results provide little support to the hypothesis that personalisation of texts improves postal questionnaire return rate compared to standard texts, in this population. There was also little evidence to suggest that personalisation led to quicker returns of questionnaires, improved questionnaire completion, or reduced the requirement for a reminder letter to elicit a response. The additional cost of personalised texts was £0.04 per participant retained.

Limitations

Eligible participants who provided a mobile phone number at enrolment to the host trial (78.4%) was lower than antipated. Nearly 30% of SWAT participants had to be excluded from analysis due to problems with text automation. Furthermore, the high proportion of returned postal questionnaires in the standard text group meant only very small improvements could ever be observed or that a ceiling effect may have been reached. Thus, a large sample size would be required in order to provide strong evidence against the null hypothesis in favour of personalisation. Together, the small sample size and high baseline event rate mean this SWAT provides limited evidence for (or against) the personalisation of texts as a means to improving retention of participants.

Conclusions

Given the uncertainty regarding the effectiveness of personalising text messages, we feel that further investigation via RCTs is warranted. Meta-analysis could be used to obtain a more precise estimate for the effectiveness of personalising texts and explore variation across different participant characteristics.

Data availability

Underlying data

Open Science Framework: OTIS Trial Text SWAT. https://doi.org/10.17605/OSF.IO/KH75X¹⁷.

This project contains the following underlying data:

OTIS_textswat_data (CSV). Underlying data associated with this study.
OTIS_textswat_data (DTA). Underlying data associated with this study.
OTIS_textswat_data_key (CSV). Key to abbreviaitons used in dataset.

Reporting guidelines

Open Science Framework: CONSORT checklist for ‘An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial’. https://doi.org/10.17605/OSF.IO/KH75X¹⁷.

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

The authors would also like to thank the embedded trial participants who returned trial questionnaires.

This paper was written by the authors on behalf of the OTIS Study Group, which is made up of the following individuals: Sophie Boyes (York Teaching Hospital NHS Foundation Trust); Sarah Cockayne (University of York); Shelley Crossland (Leicestershire Partnership NHS Trust); Avril Drummond (University of Nottingham); Caroline Fairhurst (University of York); Simon Gilbody (University of York); Catherine Hewitt (University of York); Sarah E Lamb (University of Exeter); Jennifer McCaffery (University of York); Alison Pighills (Mackay Base Hospital; Mackay Australia and James Cook University); Clare Relton (Barts and The London School of Medicine and Dentistry); Sara Rodgers (University of York); Sarah Ronaldson (University of York); Katie Whiteside (University of York), and David J. Torgerson (University of York). With the exception of Ann Cochrane (University of York), who supported data collection for the host OTIS trial, all members of the OTIS Study Group were involved in designing this SWAT.

The authors acknowledge Adwoa Parker (University of York) for pre-registering this embedded trial with the Northern Ireland Hub for Trials Methodology Research SWAT Repository (SWAT 35).

Faculty Opinions recommended

References

1. Torgerson D, Torgerson C: Designing Randomised Trials in Health, Education and the Social Sciences. Basingstoke: Palgrave MacMillan; 2008. Publisher Full Text
2. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15(1): 399. PubMed Abstract | Publisher Full Text | Free Full Text
3. Adamson J, Hewitt CE, Torgerson DJ: Producing better evidence on how to improve randomised controlled trials. BMJ. 2015; 351: h4923. PubMed Abstract | Publisher Full Text
4. Madurasinghe VW; Sandra Eldridge on behalf of MRC START Group and Gordon Forbes on behalf of the START Expert Consensus Group: Guidelines for reporting embedded recruitment trial. Trials. 2016; 17: 27. PubMed Abstract | Publisher Full Text | Free Full Text
5. Treweek S, Bevan S, Bower P, et al.: Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)? Trials. 2018; 19(1): 139. PubMed Abstract | Publisher Full Text | Free Full Text
6. Edwards PJ, Roberts I, Clarke MJ, et al.: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3): MR000008. PubMed Abstract | Publisher Full Text
7. Brueton VC, Tierney J, Stenning S, et al.: Strategies to improve retention in randomised trials. Cochrane Database Syst Rev. 2013; (12): MR000032. PubMed Abstract | Publisher Full Text | Free Full Text
8. Ashby R, Turner G, Cross B, et al.: A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011; 64(2): 208–212. PubMed Abstract | Publisher Full Text
9. Keding A, Brabyn S, MacPherson H, et al.: Text message reminders to improve questionnaire response rates. J Clin Epidemiol. 2016; 79: 90–95. PubMed Abstract | Publisher Full Text
10. Clark L, Ronaldson S, Dyson L, et al.: Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis. J Clin Epidemiol. 2015; 68(12): 1446–1450. PubMed Abstract | Publisher Full Text
11. Man MS, Tilbrook HE, Jayakody S, et al.: Electronic reminders did not improve postal questionnaire response rates or response times: a randomized controlled trial. J Clin Epidemiol. 2011; 64(9): 1001–1004. PubMed Abstract | Publisher Full Text
12. Brabyn S, Adamson J, MacPherson H, et al.: Short message service text messaging was feasible as a tool for data collection in a trial of treatment for irritable bowel syndrome. J Clin Epidemiol. 2014; 67(9): 993–1000. PubMed Abstract | Publisher Full Text
13. Richmond SJ, Keding A, Hover M, et al.: Feasibility, acceptability and validity of SMS text messaging for measuring change in depression during a randomised controlled trial. BMC Psychiatry. 2015; 15: 68. PubMed Abstract | Publisher Full Text | Free Full Text
14. Haynes LC, Green DP, Gallagher R, et al.: Collection of Delinquent Fines: An Adaptive Randomized Trial to Assess the Effectiveness of Alternative Text Messages. J Policy Analysis and Management. 2013; 32(4): 718–30. Publisher Full Text
15. Cockayne S, Pighills A, Adamson J, et al.: Can occupational therapist-led home environmental assessment prevent falls in older people? A modified cohort randomised controlled trial protocol. BMJ Open. 2018; 8(9): e022488. PubMed Abstract | Publisher Full Text | Free Full Text
16. StataCorp: Stata Statistical Software: Release 15. College Station, TX:StataCorp LLC 2017. Reference Source
17. Welch C: OTIS Trial Text SWAT. 2020. http://www.doi.org/10.17605/OSF.IO/KH75X

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 26 Feb 2020

Author details Author details

¹ York Trials Unit, University of York, York, UK

Ann Cochrane
Roles: Investigation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Charlie Welch
Roles: Data Curation, Formal Analysis, Software, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Caroline Fairhurst
Roles: Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Software, Validation, Visualization, Writing – Review & Editing

Sarah Cockayne
Roles: Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Validation, Visualization, Writing – Review & Editing

David J. Torgerson
Roles: Conceptualization, Funding Acquisition, Methodology, Resources, Supervision, Validation, Visualization, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

This study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number 14/49/149). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 26 Feb 2020, 9:154

https://doi.org/10.12688/f1000research.22361.1

Copyright

© 2020 Cochrane A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Cochrane A, Welch C, Fairhurst C et al. An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial [version 1; peer review: 2 approved] F1000Research 2020, 9:154 (https://doi.org/10.12688/f1000research.22361.1)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 26 Feb 2020

Views

13

Reviewer Report 21 Apr 2020

Phil J. Edwards, Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK

Approved

https://doi.org/10.5256/f1000research.24670.r61887

It has been shown previously that postal questionnaire response can be increased by using an SMS reminder (Edwards et al., 2009¹) and that questionnaire response can be increased by personally addressing participants by name on cover letters (Scott and Edwards, 2006²). There have, ... Continue reading

It has been shown previously that postal questionnaire response can be increased by using an SMS reminder (Edwards et al., 2009¹) and that questionnaire response can be increased by personally addressing participants by name on cover letters (Scott and Edwards, 2006²). There have, however, been few randomised trials of a personalised reminder SMS. This study is therefore a welcome addition to the literature.

This is a report of the results of a randomised controlled trial nested within an RCT of occupational therapist-led home environmental assessment for the prevention of falls in older people. The nested trial sought to evaluate the effect on questionnaire response of a personalised text message compared to a standard (non-personalised) text in this elderly population. 403 participants who had agreed to receive text communication during participation, had provided a mobile number, and were due to receive their four-month post-randomisation follow-up postal questionnaire, were randomised into this nested trial. The proportion that responded in the personalised text message group was 97.8%, lower than the proportion that responded in the non-personalised text message group (98.6%); there was thus no evidence that the intervention had any effect on questionnaire response (p=0.63). The report is a clear and concise account of the study, citing the current literature. The authors may consider it helpful to include Pippa Scott’s study in their introduction or discussion. The study design seems appropriate and the work appears to be technically sound. The conclusions are adequately supported by the results, and the study makes a useful contribution to the data collection literature.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, et al.: Methods to increase response to postal and electronic questionnaires.Cochrane Database Syst Rev. 2009. MR000008 PubMed Abstract | Publisher Full Text
2. Scott P, Edwards P: Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials.BMC Health Serv Res. 2006; 6: 111 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Evidence-based Data Collection

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

22

Reviewer Report 11 Mar 2020

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland

Approved

https://doi.org/10.5256/f1000research.24670.r60604

Summary:

This was a SWAT within the OTIS trial, an occupational therapist-led home environmental assessment for the prevention of falls in older people in the UK. The SWAT was a two-arm, parallel group study. The primary outcome ... Continue reading

Summary:

This was a SWAT within the OTIS trial, an occupational therapist-led home environmental assessment for the prevention of falls in older people in the UK. The SWAT was a two-arm, parallel group study. The primary outcome was the proportion of participants who returned the four-month follow-up postal questionnaire. Secondary outcomes were: time to response, completeness of response, requirement of a reminder letter, cost effectiveness. The conclusion was that personalised texts were not superior to standard texts in any of the outcomes assessed. This is a very well conducted study and the clarity of presentation is to be commended.

Answers to main headings for the review:

The work is clearly and accurately presented for the most part, and it does cite the relevant literature as well as the current literature. I have added some comments/corrections below to address.
The study design is appropriate and the work is technically sound. Though I understand the purpose of the study was to evaluate personalised text messages versus non-personalised messages, I would have liked to see a comparison with those that received reminder letters only. Would this be possible with the current data set?
Sufficient details of the methods and analysis are provided for the most part but I’ve asked for a few minor issues to be addressed in the comments below.
The statistical analysis and its interpretation are appropriate.
Yes, the authors have added information on data availability.
The conclusions drawn are supported by the results but they are different in the abstract to the discussion section of the paper. The authors are definitive in the abstract but are less definitive in the discussion section saying the results provide “little” support. Can we really say little support? I think this should be stronger. There is no evidence to support personalised text messages in this study. I understand there were limitations but these are the findings from this study. I’d ask the authors to address the use of the term “little” in the discussion section.

Further Comments/Corrections:

The reminder letter appears for the first time in the results section and in table 4. This needs to be detailed in the methods section. What justified a person being sent a reminder letter/why were only 21 reminder letters sent? Also, you should add the reminder letter to Figure 1. I expect that their inclusion didn’t affect the results given the small numbers and equal proportions in both groups, however, a re-run of the analysis excluding them, and a sentence to say that it didn’t affect the results.
Rather than just present the actual costs of each type of text message in the results section under “cost-effectiveness”, can you present the findings of the analysis from both? I appreciate it’s presented in Table 5 but comment on it.
The outcome definition in Table 2 “proportion of questionnaires returned” should be adjusted as its current meaning does not match your intention. It is not the proportion of questionnaires returned to YTU at four months post-randomisation, it is the proportion of questionnaires returned or the proportion of questionnaires distributed at the four-month randomisation period that were returned or as you have in your main text, the proportion of participants who returned their four-month postal questionnaire. It would be helpful if you specified here how long you gave them to return it. I would add…"within x days".
Paragraph two of the results section, "…compared with 142 in the standardised text group….” - replace standardised with standard.
In Table 5, the heading cost of texts is misleading as it implies that more than one text was sent to each person. Change to “cost of text”.

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Epidemiology and Clinical Trial Methodology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 26 Feb 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 1 26 Feb 20	read	read

Frances Shiely, University College Cork, Cork, Ireland
Phil J. Edwards, London School of Hygiene and Tropical Medicine, London, UK

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

13 Views

21 Apr 2020 | for Version 1

Phil J. Edwards, Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK

13 Views Cite this report Responses(0)

Approved

It has been shown previously that postal questionnaire response can be increased by using an SMS reminder (Edwards et al., 2009¹) and that questionnaire response can be increased by personally addressing participants by name on cover letters (Scott and Edwards, 2006²). There have, however, been few randomised trials of a personalised reminder SMS. This study is therefore a welcome addition to the literature.

This is a report of the results of a randomised controlled trial nested within an RCT of occupational therapist-led home environmental assessment for the prevention of falls in older people. The nested trial sought to evaluate the effect on questionnaire response of a personalised text message compared to a standard (non-personalised) text in this elderly population. 403 participants who had agreed to receive text communication during participation, had provided a mobile number, and were due to receive their four-month post-randomisation follow-up postal questionnaire, were randomised into this nested trial. The proportion that responded in the personalised text message group was 97.8%, lower than the proportion that responded in the non-personalised text message group (98.6%); there was thus no evidence that the intervention had any effect on questionnaire response (p=0.63). The report is a clear and concise account of the study, citing the current literature. The authors may consider it helpful to include Pippa Scott’s study in their introduction or discussion. The study design seems appropriate and the work appears to be technically sound. The conclusions are adequately supported by the results, and the study makes a useful contribution to the data collection literature.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, et al.: Methods to increase response to postal and electronic questionnaires.Cochrane Database Syst Rev. 2009. MR000008 PubMed Abstract | Publisher Full Text
2. Scott P, Edwards P: Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials.BMC Health Serv Res. 2006; 6: 111 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Evidence-based Data Collection

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

22 Views

11 Mar 2020 | for Version 1

Frances Shiely, School of Public Health, University College Cork, Cork, Ireland

22 Views Cite this report Responses(0)

Approved

Summary:

This was a SWAT within the OTIS trial, an occupational therapist-led home environmental assessment for the prevention of falls in older people in the UK. The SWAT was a two-arm, parallel group study. The primary outcome was the proportion of participants who returned the four-month follow-up postal questionnaire. Secondary outcomes were: time to response, completeness of response, requirement of a reminder letter, cost effectiveness. The conclusion was that personalised texts were not superior to standard texts in any of the outcomes assessed. This is a very well conducted study and the clarity of presentation is to be commended.

Answers to main headings for the review:

The work is clearly and accurately presented for the most part, and it does cite the relevant literature as well as the current literature. I have added some comments/corrections below to address.
The study design is appropriate and the work is technically sound. Though I understand the purpose of the study was to evaluate personalised text messages versus non-personalised messages, I would have liked to see a comparison with those that received reminder letters only. Would this be possible with the current data set?
Sufficient details of the methods and analysis are provided for the most part but I’ve asked for a few minor issues to be addressed in the comments below.
The statistical analysis and its interpretation are appropriate.
Yes, the authors have added information on data availability.
The conclusions drawn are supported by the results but they are different in the abstract to the discussion section of the paper. The authors are definitive in the abstract but are less definitive in the discussion section saying the results provide “little” support. Can we really say little support? I think this should be stronger. There is no evidence to support personalised text messages in this study. I understand there were limitations but these are the findings from this study. I’d ask the authors to address the use of the term “little” in the discussion section.

Further Comments/Corrections:

The reminder letter appears for the first time in the results section and in table 4. This needs to be detailed in the methods section. What justified a person being sent a reminder letter/why were only 21 reminder letters sent? Also, you should add the reminder letter to Figure 1. I expect that their inclusion didn’t affect the results given the small numbers and equal proportions in both groups, however, a re-run of the analysis excluding them, and a sentence to say that it didn’t affect the results.
Rather than just present the actual costs of each type of text message in the results section under “cost-effectiveness”, can you present the findings of the analysis from both? I appreciate it’s presented in Table 5 but comment on it.
The outcome definition in Table 2 “proportion of questionnaires returned” should be adjusted as its current meaning does not match your intention. It is not the proportion of questionnaires returned to YTU at four months post-randomisation, it is the proportion of questionnaires returned or the proportion of questionnaires distributed at the four-month randomisation period that were returned or as you have in your main text, the proportion of participants who returned their four-month postal questionnaire. It would be helpful if you specified here how long you gave them to return it. I would add…"within x days".
Paragraph two of the results section, "…compared with 142 in the standardised text group….” - replace standardised with standard.
In Table 5, the heading cost of texts is misleading as it implies that more than one text was sent to each person. Change to “cost of text”.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology and Clinical Trial Methodology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

[1] 1. Torgerson D, Torgerson C: Designing Randomised Trials in Health, Education and the Social Sciences. Basingstoke: Palgrave MacMillan; 2008. Publisher Full Text

[2] 2. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15(1): 399. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Adamson J, Hewitt CE, Torgerson DJ: Producing better evidence on how to improve randomised controlled trials. BMJ. 2015; 351: h4923. PubMed Abstract | Publisher Full Text

[4] 4. Madurasinghe VW; Sandra Eldridge on behalf of MRC START Group and Gordon Forbes on behalf of the START Expert Consensus Group: Guidelines for reporting embedded recruitment trial. Trials. 2016; 17: 27. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Treweek S, Bevan S, Bower P, et al.: Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)? Trials. 2018; 19(1): 139. PubMed Abstract | Publisher Full Text | Free Full Text

[6] 6. Edwards PJ, Roberts I, Clarke MJ, et al.: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3): MR000008. PubMed Abstract | Publisher Full Text

[7] 7. Brueton VC, Tierney J, Stenning S, et al.: Strategies to improve retention in randomised trials. Cochrane Database Syst Rev. 2013; (12): MR000032. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Ashby R, Turner G, Cross B, et al.: A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011; 64(2): 208–212. PubMed Abstract | Publisher Full Text

[9] 9. Keding A, Brabyn S, MacPherson H, et al.: Text message reminders to improve questionnaire response rates. J Clin Epidemiol. 2016; 79: 90–95. PubMed Abstract | Publisher Full Text

[10] 10. Clark L, Ronaldson S, Dyson L, et al.: Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis. J Clin Epidemiol. 2015; 68(12): 1446–1450. PubMed Abstract | Publisher Full Text

[11] 11. Man MS, Tilbrook HE, Jayakody S, et al.: Electronic reminders did not improve postal questionnaire response rates or response times: a randomized controlled trial. J Clin Epidemiol. 2011; 64(9): 1001–1004. PubMed Abstract | Publisher Full Text

[12] 12. Brabyn S, Adamson J, MacPherson H, et al.: Short message service text messaging was feasible as a tool for data collection in a trial of treatment for irritable bowel syndrome. J Clin Epidemiol. 2014; 67(9): 993–1000. PubMed Abstract | Publisher Full Text

[13] 13. Richmond SJ, Keding A, Hover M, et al.: Feasibility, acceptability and validity of SMS text messaging for measuring change in depression during a randomised controlled trial. BMC Psychiatry. 2015; 15: 68. PubMed Abstract | Publisher Full Text | Free Full Text

[14] 14. Haynes LC, Green DP, Gallagher R, et al.: Collection of Delinquent Fines: An Adaptive Randomized Trial to Assess the Effectiveness of Alternative Text Messages. J Policy Analysis and Management. 2013; 32(4): 718–30. Publisher Full Text

[15] 15. Cockayne S, Pighills A, Adamson J, et al.: Can occupational therapist-led home environmental assessment prevent falls in older people? A modified cohort randomised controlled trial protocol. BMJ Open. 2018; 8(9): e022488. PubMed Abstract | Publisher Full Text | Free Full Text

[16] 16. StataCorp: Stata Statistical Software: Release 15. College Station, TX:StataCorp LLC 2017. Reference Source

[17] 17. Welch C: OTIS Trial Text SWAT. 2020. http://www.doi.org/10.17605/OSF.IO/KH75X

An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial

Abstract

Keywords

Introduction

Methods

Design

Participants

Intervention

Table 1. Text message content by allocation.

Outcomes

Table 2. SWAT primary and secondary outcomes.

Sample size

Randomisation

Blinding

Ethical approval

Statistical analysis

Table 3. Baseline characteristics of the participants included in the analysis.

Results

Figure 1. The flow of participants through the embedded trial.

Table 4. Analysis of binary outcomes.

Table 5. Costs per participant of retention at four months, by allocation and overall.

Figure 2. Kaplan-Meier curve for time to questionnaire return.

Cost-effectiveness

Discussion

Limitations

Conclusions

Data availability

Underlying data

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated