Skip to main content

Participation of adults with disorders/differences of sex development (DSD) in the clinical study dsd-LIFE: design, methodology, recruitment, data quality and study population

Abstract

Background

dsd-LIFE is a comprehensive cross-sectional clinical outcome study of individuals with disorders/differences of sex development (DSD). This study focuses on various rare genetic conditions characterized by impaired gonadal or adrenal functionality.

Methods/Design

The study aims to assess quality of life (QoL) as a measure of psychosocial adaptation, psychosexual and mental health aspects as major outcomes. Health status and functioning, medical and surgical therapies, participants’ views on health care, psychological and social support, sociodemographic factors and their interrelations will be investigated as factors associated with the outcomes. In addition, ethical considerations in the field of DSD are addressed and previous experiences with health care were gathered. One thousand and forty participants with different DSD conditions were recruited by 14 study centres in 6 European countries (France, Germany, the Netherlands, Poland, Sweden and the United Kingdom) from February 2014 until September 2015. The conditions included were: Turner syndrome (n = 301); 45,X0/46,XY conditions (n = 45); Klinefelter syndrome (n = 218); 47,XYY (n = 1); 46,XY gonadal dysgenesis/ovotestes (n = 63); complete androgen insensitivity (CAIS) (n = 71); partial androgen insensitivity (PAIS) (n = 35) and androgen synthesis disorders (n = 20); severe hypospadias (n = 25); other or non-classified 46,XY DSD (n = 8); 46,XX congenital adrenal hyperplasia (CAH) (n = 226); 46,XX gonadal dysgenesis/ovotestis (n = 21); and 46,XX in males (n = 6). For an add-on study, 121 46,XY male-assigned individuals with CAH due to 21-hydroxylase deficiency were recruited. Mean age of participants’ was 32.4 (+/− 13.6 years).

Discussion

Participation was high in conditions not commonly described as DSD, such as Turner and Klinefelter syndromes or CAH. Recruitment of individuals with XY DSD conditions proved to be more difficult. The data collection of PROs resulted in high data quality. Within medical and physical examination data, more missings and/or inaccurate data were found than expected. The European dsd-LIFE study recruited and evaluated the largest cross-sectional sample of individuals with different conditions classified under the term DSD. The data from this large sample will provide a sufficient basis for evidence-based recommendations for improvement of clinical care of individuals affected by a DSD condition.

Trial registration

German Clinical Trials Register DRKS00006072.

Peer Review reports

Background

Rare conditions – Disorders/differences in sex development

“Disorders/differences of sex development (DSD)” is used as an umbrella term for various rare conditions that are characterized by an incongruence of chromosomal, gonadal and genital sex development. The approximate incidence of genital anomalies is estimated to occur 1:4500 live births [1].

The term DSD and a new system for nomenclature were introduced by the Chicago Consensus Group in 2005, replacing previous nomenclature that was perceived negatively by affected individuals. The classification of DSD distinguishes three major groups: (1) DSD with atypical sex chromosome configurations, including Turner syndrome (45,X0 and mosaicisms), Klinefelter syndrome (47,XXY and mosaicisms) and conditions with 45,X/46,XY or 46,XX/46,XY karyotypes; (2) XY DSD, encompassing conditions characterized by 46,XY karyotype and impairment of testicular development, androgen biosynthesis or action, AMH biosynthesis or action, hypospadias, cloacal exstrophy and other syndromic forms; (3) XX DSD, comprising conditions characterized by 46,XX karyotype and androgen excess, such as congenital adrenal hyperplasia, P450 oxidoreductase deficiency, aromatase deficiency or exogenous causes, impairment of ovarian, uterine or vaginal development and other syndromic forms [1]. The multitude of mechanisms related to sex determination and sex differentiation—such as genes involved in gonadal development, androgen receptor function and steroid biosynthesis—makes the diagnoses of the XY DSD group complex and challenging. The heterogeneity of clinical presentations of DSD conditions results in variations of treatment options, including medical, surgical or psychotherapeutic, while their rarity carries the consequence of limited ad hoc knowledge about those various options [2,3,4,5,6]. Clinical studies on DSD are often single-centre experiences from regional samples with a limited number of unambiguous diagnoses of participants, and many of the studies lack appropriate comparison groups. The comparability of these studies is often impeded because different definitions and study methodologies are used [7,8,9,10,11,12,13,14,15,16,17,18,19].

The study dsd-LIFE, funded by the European Commission (7th Framework Programme, FP7), has sought to overcome some of the above-mentioned challenges by using a standardized and consistent study methodology in a large number of individuals with DSD conditions in six European countries.

Methods/Design

The dsd-LIFE study

dsd-LIFE investigates and compares the long-term outcomes of surgical and hormonal therapy and psychological and social support in adequate numbers of adolescents (from the age of 16) and adults with DSD conditions, aiming to provide the basis for improvements in evidence-based recommendations for care. The long-term impact of the study is expected to result in improvement of care and subsequently higher overall quality of life (QoL) and better integration and participation of individuals with DSD in society. To reach these aims, we assessed as main outcomes QoL, psychosexual and mental health aspects. As determinants we collected objective data on sociodemographic factors, social participation, religion, culture, physical and mental health, risk for mental health such as anxiety, depression, autism and ADHD, treatments such as genital surgery and hormone therapy, fertility, psychological and social support, current and past health care, psychological factors such as self-esteem, coping, autonomy, attachment and subjective appraisal of relationships as well as appraisal of relations and personal’ views on ethics of care (Fig. 1).

Fig. 1
figure 1

Conceptual model

The dsd-LIFE consortium consists of 16 European partners from Germany, France, the Netherlands, Poland, Sweden and the United Kingdom (UK). Fourteen of these partners acted as recruiting sites, including Berlin, München, Lübeck, Münster (D), Paris, Lyon Montpellier and Toulouse (F); Amsterdam and Nijmegen (NL); Lodz and Warszawa (P); Stockholm (S); and Birmingham (UK) (Fig. 2). In addition to the recruiting sites, the consortium included one centre with expertise in medical ethics (Göttingen) and one partner with expertise in European community (EC) project management (München). The Scientific Advisory Board consisted of international experts in the fields of endocrinology, psychology, surgery, and ethics and a representative of a longstanding patient support group for androgen insensitivity syndrome. An important scientific basis for this study was the results of a German network DSD study published in 2009, which was the largest clinical evaluation study of DSD available at the time [20].

Fig. 2
figure 2

Locations of project partners. The following centres participated in the study: France: Université Claude Bernard Lyon; Le Centre Hospitalier Universitaire Montpellier; Université Paris-Sud, Paris; Le Centre Hospitalier Universitaire de Toulouse. Germany: Charité Universitätsmedizin Berlin; Ludwig-Maximilians-Universität, München; University of Lübeck; Universitätsmedizin Göttingen; Westfälische Wilhelms-Universität Münster. Poland: Medical University of Lodz; Children’s Memorial Health Institute, Warszawa. Sweden: Karolinska Institutet, Stockholm. United Kingdom: University of Birmingham. The Netherlands: VU University Medical Center, Amsterdam; Radboud University Nijmegen Medical Center, Nijmegen

In this paper, we outline the theoretical and methodological framework of the study. We compare the study aims related to participation rates and data quality to the actual sample recruited and the information that became available from various sources.

Conceptual model, study design and methods

Conceptual model

Historically, clinicians, administrators, researchers, and policy makers have evaluated the effectiveness of treatments and interventions with little input from patients. With the rising participation of patients and clients in shared decision making, evaluation of the impact of care on day-to-day life has become increasingly important. When validated questionnaires are administered directly to participants without involving clinicians, they are called patient-reported outcomes (PROs) and may include reports on objective parameters such as sociodemographic factors, measures of functioning and health status as well as subjective appraisal of one’s QoL and other relevant aspects of well-being. Conceptually, dsd-LIFE developed a framework to determine the major outcome QoL solely from the subjective perspective of the participant and adopted the definition of health proposed by the World Health Organization (WHO), namely, physical, psychological and social well-being (Fig. 1) [21, 22]. For each dimension of QoL, dsd-LIFE sought to collect information from at least one additional perspective. For physical health, dsd-LIFE aimed to collect data from clinicians through medical histories, clinical examinations and data documented in medical records. For psychological health, we probed for indicators of special health care needs, clinical diagnoses and screening instruments of mental health conditions. For social well-being, we collected information on sociodemographics, participation and inclusion [21, 22]. For satisfaction with health care, we measured the subjective rating of the participants of the care they had received, such as satisfaction with medical and psychological care, medical information management, patient-centred care and doctors’ behaviour. To relate outcomes to treatments, we collected medical data of current and past treatments and interventions from medical files as available. The overall aim of the study was to improve health care in the clinical population of individuals with DSD conditions.

Study design

dsd-LIFE was designed as a non-interventional, clinical, cross-sectional study. The inclusion criteria covered all conditions described as DSD in the classification system of the Chicago Consensus Conference and an age of 16 years or older. Verification of the correct diagnosis was conducted by examining clinical medical and genetic data; all participants had to be seen at least once at one of the recruiting sites. Health status was assessed by a physical examination, potentially augmented by further examinations. Current and past treatment data were obtained from a medical interview and medical chart review. The wide age range of the subjects allowed assessment and comparison of treatments that were typical in certain decades but changed over time. PRO data came mainly from an online survey, allowing privacy and confidentiality for all participants. The design was chosen to allow the analysis of associations of treatments and interventions experienced in the past with contemporary QoL, physical and mental health, psychosexual functioning and satisfaction with care. In addition, we sought to examine participants’ views and perspectives on the ethics of general health care and controversial treatments (Fig. 3).

Fig. 3
figure 3

Study design and work packages of dsd-LIFE

We did not recruit a comparison or control group, but results of major outcomes will be compared to reference data. Moreover, we planned to use group comparisons of the diagnoses within the sample because some issues such as e.g. fertility or psychosexual problems are, to varying degrees, issues for all people affected by DSD.

Instruments

The use of available, standardized instruments with reference data from the general population was preferred whenever possible, thus allowing comparisons to the general population, specifically in the areas of sociodemographics, QoL and screening measures of physical and mental health, psychosexual outcome, psychological factors and overall satisfaction with health care. For specific aspects of DSD, we included “self-constructed” items. The iterative review of the suggested instruments and measurements was performed by experienced patient support group representatives and by the project’s Advisory Board.

For application of the final questionnaire in all study-relevant languages (Dutch, English, French, German, Polish, and Swedish), professional translation of the instruments used was required. To some extent, the standardized instruments chosen were available in the languages needed (Table 1). For those instruments and (self-constructed) items that required translation, we followed the linguistic validation process of the international translation guidelines [23, 24].

Table 1 Instruments used in dsd-LIFE

Medical interview, retrospective chart review and medical examinations (part 1)

Generally, the data came from a personal medical review on contemporary signs and symptoms, past medical history and a physical examination with a physician; the data were input by a study nurse or physician (case report form; CRF):

  1. 1.

    The medical interview (239 items) collected standardized information on the diagnosis, karyotype and comorbidities of the participants and their families, physical activity, smoking behaviour, current therapy, application, therapy monitoring, previous hormone therapies and psychological counselling and menstruation.

A retrospective chart review of past diagnosis and medical treatment was performed to obtain information from medical records, including birth history and genital phenotype at diagnosis, date of diagnosis, and hormones and hormone therapy at diagnosis (dosage, application) (79 items). Previous surgical procedures and complications were also retrieved from medical records (29 items). Further retrospective data related to hormone treatment were gathered from medical records at the time of puberty (Tanner stage II) (6 items) and completion of puberty (> = 16 years) (45 items). For individuals with congenital adrenal hyperplasia (CAH), additional data on hormone therapy were taken from the records at the ages of 9 months and 6 years to evaluate steroid hormone levels and treatment outcomes (30 items each). Moreover, descriptions of ambiguity of genitalia (44 items) were gathered from the records, if applicable.

  1. 2.

    The general physical exam was used to obtain a general impression of the patient; anthropometric measures of height, weight, body mass index (BMI), and hip and waist circumference; and blood pressure (mean of 3 measurements). The exam also revealed the presence of acanthosis; stretch marks; acne; hirsutism; decrease of body hair; bruised skin; and pubic hair. Breast development was assessed according to the Tanner and Ferriman Galway score (13 items).

  2. 3.

    An optional gynaecological/urological examination, including a gynaecological history if applicable, general genital development and surgical results, was performed for participants who had undergone genital constructive surgery (11 gynaecology/9 urology items).

  3. 4.

    The laboratory investigations featured blood tests that used a fasting blood sample to measure metabolic parameters such as lipids, glucose, insulin, R-Homa, liver enzymes, uric acid, whole blood count, and renal parameters for all diagnostic groups; hormones in all females: gonadotropins, oestrogen, androgens, and thyroid function; hormones in all males: gonadotropins, oestrogen, androgens, anti-muellerian hormone (AMH), inhibin B and thyroid function; and hormones in all participants with CAH, including 17-hydroxyprogesterone, androstenedione, and renin. In participants with gonads in place, AMH, inhibin B and gonadal tumour markers were measured (38 items).

  4. 5.

    The optional technical assessments included dual-energy X-ray absorptiometry (Dexa-scans) measuring bone mineral density and body composition (15 items); bioelectrical impedance analysis (BIA) evaluating body composition (12 items); ultrasound of the carotic artery measuring intima media thickness (IMT) (9 items); ultrasound of the uterus and ovaries estimating uterine size to evaluate the development and oestrogen effects and the ovarian structure and possible abnormalities; testicular ultrasound measuring the testicular size and structure and possible abnormalities (57 items); and a spermiogramme evaluating sperm quality (count, motility, morphology, vitality, and volume) (6 items).

Patient-related outcomes (PROs, part 2)

This part of the study included standardized instruments and self-constructed questionnaires (Table 1).

  1. 1.

    Sociodemographic data such as age, area of dwelling, education, household and family, occupation, nationality, religious denomination and social participation were gathered according to the European Social Survey (ESS; source questionnaire amendment 01, 2012/2013). All questions were provided in interview format to maximize the accuracy and objectivity of the information. For dsd-LIFE, we transformed these questions into self-reporting questionnaires, retaining the response format. Validated translations and reference data are available for all countries of the dsd-LIFE consortium (http://www.europeansocialsurvey.org).

  2. 2.

    For the assessment of QoL, we used the WHOQOL-BREF, which evaluates physical and psychological QoL, social relationships, environment and global QoL; reference data on both the general population and clinical samples for comparison are available [25, 26].

  3. 3.

    To assess physical health we used questions from the European Health Interview Survey (EHIS) and self-constructed condition specific questions.

  4. 4.

    To evaluate mental health we used questions from the EHIS and self-constructed questions. We used the Hospital Anxiety and Depression Scale (HADS) to screen for depression and anxiety. We used the Autism Spectrum Quotient (AQ)-10 test and the Adult ADHD Self-Report to screen for signs and symptoms of for autism and ADHD as they have been described as common comorbidities in the DSD population in previous studies [27, 28].

  5. 5.

    To asses psychological factors the following questionnaires were chosen: the Rosenberg Self-esteem Scale (RSES) for self-esteem, a self-constructed questionnaire for shame, stigma and coping, questions form the ESS for autonomy, the Relationship Structures (ECR-RS) Questionnaire for attachment.

  6. 6.

    Psychosexual issues were evaluated by standardized and condition-specific self-contructed questionnaire. The Body Image Scale (BIS) was chosen to investigate body satisfaction, an issue of high relevance in the condition groups studied [29, 30].

    We opted to evaluate gender dysphoria with the well-established and relatively short Utrecht Gender Dysphoria Scale (UGDS). This instrument has been developed for the assessment of people without DSD who have gender identity disorder but has been used in previous studies in DSD [31]. In addition, feelings of male- and femaleness were evaluated by two self-constructed questions. We chose not to evaluate gender role behaviour, which shows a wide variation in the normal population and does not have clinical relevance in itself.

    The questionnaire on sexuality was developed for the German network study evaluating developmental milestones sexual problems, feelings and satisfaction with sexual life. The items had performed positively in that study, and the scale was thus selected and translated into the various languages for dsd-LIFE [10, 32]. Sexual orientation was measured with the Kinsey scale, and the wording was changed to make the questions less dichotomous in terms of gender [33, 34].

  7. 7.

    For evaluation of satisfaction with health care, the Client Satisfaction Questionnaire (CSQ4) and an adapted version of the Child Health Care – Satisfaction, Utilization, and Needs Questionnaire (CHC-SUN) was chosen. The CHC-SUN was originally developed as a proxy-report version for parents of children with chronic conditions and later adapted and tested as a self-report version for young adults with chronic diseases. For this study, an extended short form of the CHC-SUN was applied, and several self-constructed items on satisfaction with care and needs were added [35, 36]. For assessment of current and past health care, satisfaction with information management, satisfaction with hormone therapies, genital surgery, current and previous treatment/psychological/social support and fertilty self-constructed questions were used.

  8. 8.

    Participants’ perspectives on ethical issues were assessed by self-constructed questionnaires. The items on ethical issues had been developed by the ethical research group in Göttingen based on qualitative work with focus groups and individual interviews. The interview questions were closely related to the recommendations for care of individuals with DSD, which were developed by the German Ethical Council in 2012 [37, 38]

Obligatory requirements for participation in the study were participation in the medical clinical interview and completion of the PRO questionnaire. Medical examinations were considered an important part of the study but were not obligatory. Participants not interested in medical examinations were included into the study if they met the above requirement, if they had visited the centre in person at least once, and if they had documented previous clinical consultations and diagnosis (Fig. 4).

Fig. 4
figure 4

Study procedure

Inclusion criteria

The study included individuals with a confirmed diagnosis of DSD with variants in sex chromosomes, including Turner syndrome, Klinefelter syndrome and mixed gonadal dysgenesis (45,X/46,XY); XY conditions, including testicular dysgenesis, impairment of testosterone synthesis or action and hypospadias; XX conditions, including ovarian dysgenesis, congenital adrenal hyperplasia (CAH), and XX males; and ovotesticular DSD. XY male participants with CAH (21-OHD and 11ß-HSD deficiency) were included, although, according to the definition, they do not belong to the DSD classification. This group was an add-on study population because it was hypothesized that in many ways these subjects might face similar problems as persons within the DSD classification, such as e.g. sex hormone imbalances and fertility problems. Participants had to be at least 16 years old with a medically confirmed clinical and/or genetic diagnosis (Fig. 4). The age limit of 16 years was chosen because pubertal development is essential for the evaluation of psychosexual development. Moreover, the initial effects of treatment in childhood and adolescence on the metabolic system can be evaluated at the beginning of adulthood. In addition, the perspectives of adolescents regarding their DSD condition and treatment are of particular interest.

Exclusion criteria

Individuals not fulfilling the diagnostic criteria or incapable of giving consent or answering the questions on their own were excluded. Subjects with Mayer-Rokitansky-Küster-Hauser syndrome (uterus agenesis) and other complex non-endocrine urogenital malformations were also excluded as most centers had no access to patients with these diagnoses.

Ethics

Ethical approval was first sought from the medical ethics committee at the Charité Universitätsmedizin Berlin. Ethical approvals were subsequently sought from the other study centres. Data protection and valid procedures to secure the anonymity of the participants were crucial requirements of the study and for the ethical approval. Dsd-LIFE participants were informed thoroughly about the study content, aims and potential risks and subsequently gave written informed consent. The informed consent was written in lay language to ensure the understandability of the aims, procedures and potential risks of participation. All participants had to consent to the release of their medical data retrieved from medical charts, including genetic test results. If the participant was underage, both the potential participant and the parents of the participant had to give written informed consent.

The opportunity for withdrawal from the study was possible at any stage of the study. Travel costs of the participants were covered upon request. According to the World Medical Association’s Helsinki Declaration and the Council of European Convention on Human Rights and Biomedicine [39], dsd-LIFE posed minimal risk for the participants (e.g., medical exams, blood sampling) [40, 41].

Recruitment

The 14 recruiting centres approached former and current patients with a diagnosis meeting the inclusion criteria for dsd-LIFE and promoted participation in the study in various additional ways. The means of contacting potential study participants included regular mail, e-mail, phone, direct contact of the physician (at consultation hours at the recruitment site or at consultation hours of regional endocrinologists linked to the recruitment site), patient support groups and information from the dsd-LIFE website (www.dsd-life.eu). A standardized record was kept by each centre to document the recruitment procedure and the informed consent process. In this way, the process of individual approach and information about acceptance or denial of participation and inclusion were retained. Participant recruitment started on February 1, 2014 and ended on September 30, 2015. To obtain adequate numbers in the XY DSD group, recruitment for this group was prolonged from 16 to 20 months.

Study logistics

Each recruiting centre established a team to contact, include and monitor participants throughout the study. Study teams could consist of endocrinologists, gynaecologists, urologists, surgeons, psychologists, and study nurses: usually, the endocrinologist informed the potential participant, assured or verified consent and subsequently performed the medical examination. Gynaecologists, urologists or surgeons accomplished the gynaecological or urological examinations as required and if the participant had consented to these examinations. All clinicians provided counselling for treatment options if needed or arranged for appropriate referrals. The study nurse or physician entered the data of the medical CRFs.

The PRO was explained by the study nurse. To ensure confidentiality and unbiased responses, participants were asked to respond to the online version of the PRO, which was accessible only with a secure password in the recruitment centres. A physician, and when possible a psychologist, was available for questions and to support the participant at any time during the study. In case the participant was not able or willing to complete the PRO at the recruitment centre, online access was provided for answering the questions at home. If needed, a paper-and-pencil version was provided for the participant, and data entry was accomplished by the study nurse after the patient had completed the questions.

Training and quality management

All of the study procedures, beginning with enrolment, were set up according to standard operation procedures (SOP). Each recruitment team received digital and printed booklets with the SOPs and was trained before recruitment started. The training occurred either face-to-face in a local study centre or via Skype conferences. The training and the SOPs addressed all relevant study procedures, including the inclusion or exclusion criteria; organizational aspects (patient information, informed consent, pseudonymization of data, maintenance of study records, patient time plan, travel reimbursement, contact details for participants); standardized performance of medical exams; blood sampling and biobanking; registration of core data on the diagnosis in the international DSD patient registry, if the participant consented (I-DSD, https://www.i-dsd.org/); administration of PROs and CRFs; data entry; and handling of information associated with exclusion from the study or the decision not to approach a potential participant. Such information was held at the discretion of the recruiting centre. For ethical reasons and data protection, we were not allowed to enter data on diagnosis, age or gender or other information available from the charts in the dsd-LIFE database if the patient had not consented to participate in the study. Therefore, data on non-responders are not available in a systematic manner. To ensure ongoing recruitment and timely study performance and to ensure data quality at data entry, the study management (Work Package 2, Fig. 4) liaised regularly with each recruitment centre.

Data management, safety and quality

The data collected during the study were entered into a central database using electronic data capture. The data were entered remotely at the sites using electronic CRFs that also allowed definition of specific roles (clinical investigator, study assistant, etc.) to control access rights depending on the function within the study. To ensure data safety and data privacy, personal data were pseudonymized. Only the study data of the participants were stored in the study database. Personal data of the individuals were not saved in the database at any time. The study data could be linked to the personal data only by using the pseudonyms, which were created automatically by the system. A pseudonymization list was safely stored in the local study centres and could be accessed only by the local principal investigator. Data checks were performed at regular time points (every 3–6 months) during recruitment. At the end of data entry, the filling status of medical data and the participant questionnaires were checked. Data plausibility, data consistency, and the possibility of missing data were verified. In addition, checks were performed to identify entire empty datasets, non-empty datasets with missing data and other implausibilities and inconsistencies. Queries regarding possible data entry errors or missing values were sent to the clinical sites for clarification or completion, respectively.

The Coordinating Centre for Clinical Studies at the Charité (Koordinierungszentrum für klinische Studien, KKS Charité) performed data management and quality assurance.

All diagnoses were verified at the end of recruitment by the leading study centre at Charité Universitätsmedizin Berlin. The complex diagnoses included in the XY DSD and the CAH groups (B.K., M.S.-C., H. C.-G., A.N.) were double checked by two endocrinologists independently. Implausibilities were discussed with the local study centre.

Statistical analysis of the dsd-LIFE cohort

To describe the dsd-LIFE study group in detail, we first compared the planned, recruited and participating numbers of participants, in total and per diagnostic group (Fig. 4). The recruitment of individuals was described using absolute and relative frequencies of contacted persons and participants in total and per country. Information on “recruitment ways” (the ways potential participants were contacted) were collected from the participating sites; however, due to missing data, it was not possible to provide a comprehensive quantitative analysis. Relevant participant characteristics of the recruited study population were analysed descriptively in total and per country using means and standard deviations for continuous parameters and absolute and relative frequencies for categorical parameters. Relevant social characteristics of the dsd-LIFE cohort were compared with the results from the population-based European Social Survey for the countries involved in dsd-LIFE comparison samples [42].

To assess data quality, the number of non-responses to entire questionnaires and sub-domains was examined by calculating relative frequencies in the total cohort and in the diagnostic sub-groups. Participation in the medical examinations and the availability of retrospective data were assessed using relative frequencies per diagnosis group, per country and per gender where appropriate. All computations were performed using R (Version 3.2.2) and statistical analysis system (SAS Version 9.4).

Results

Description of study population

Recruitment

At the beginning of the project, the recruitment sites reported a total number of 2519 persons with the different DSD diagnoses currently treated in the centres or who had been treated in the past. Based on that number and considering loss to follow-up, exclusion criteria and those who declined to participate, the centres estimated 1697 potential participants in 2012 (Fig. 3). However, in the project recruitment phase (2/2014–9/2015), the centres were able to identify and to invite 3217 potential participants from their records, and 1161 agreed to participate in the study (36.1%). Most of the 1161 dsd-LIFE participants were recruited in France (n = 311, 26.8%), followed by Germany (n = 291, 25.1%), the Netherlands (n = 265, 22.8%), Sweden (n = 131, 11.3%), Poland (n = 108, 9.3%), and the United Kingdom (n = 55, 4.7%) (Table 2).

Table 2 Participation rate in dsd-LIFE

Diagnosis

Among the 1161 participants, 1040 had a DSD condition according to the Chicago classification (Table 3), and 121 were males with 21-hydroxylase deficiency or 11-β-hydroxylase deficiency. Five participants had to be excluded during the verification process of diagnoses as non-cases. Overall, the project achieved a participation rate of 36.1%, ranging from 54.4% in Sweden to 30.0% in the Netherlands. An overview of participation by country is given in Table 2. From 3217 approached persons, the recruitment means and respective participation rate could be tracked for 1717 individuals. The numbers of contacted persons/numbers of participants/participation rate were as follows: by mail: 1222/268/22%, e-mail: 40/25/63%; telephone: 91/40/44%; personal contact in the clinic: 282/215/76%; and support groups: 82/20/24%. Of these methods, response rates (76%) were highest among persons who were contacted directly by their physician. The recruitment of individuals with a diagnosis of sex chromosome DSD was achieved as intended, whereas recruitment of individuals with 46,XY DSD diagnosis and 46,XX gonadal dysgenesis diagnosis resulted in approximately half the initially planned numbers. The diagnoses and gender and of the cohort are described in detail in Table 3. Proportions of diagnoses and gender are shown in Table 6. The participants with Turner syndrome were classified according the Mortensen classification [43]. In the groups of participants with Klinefelter and Turner syndrome, the clinical diagnosis was confirmed genetically in almost all cases (>96%). In the heterogeneous groups of diagnoses summarized under XY DSD and XX CAH, the proportions of genetically confirmed diagnoses were 47.5% and 69%, respectively. However, the precise mutation responsible for the genetic conditions was not documented in the patient files in some centres due to insufficient or lack of transmission from the genetic laboratory. (In some countries, sharing genetic information with clinicians requires written informed consent by the patient.) In the group of XX gonadal dysgenesis, only clinical diagnoses were documented. Detailed information on the proportions of genetic and clinical diagnoses is shown in Table 4. In participants with CAH due to 21-hydroxylase deficiency (21-OHD), the severity of the enzyme defect was classified according to the clinical phenotype. In addition, this group was classified according to genotype because a valid genotype-phenotype correlation could be demonstrated. For 21-OHD, genotype-phenotype classification was established ranging from 0 for salt-wasting for the severest forms to C for non-classical, which is the least severe form [44, 45]. This classification according to the genotype of participants with 21-OHD deficiency is presented in Table 5.

Table 3 Classification of participants according to the Chicago classification
Table 4 Number of genetically and clinically confirmed diagnoses of XY DSD and CAH
Table 5 Classification of individuals with 21-OHD (CYP21A2) and other CAH according to genotype

Sociodemographics

The sociodemographic characteristics of the 1040 participants with a DSD diagnosis in the dsd-LIFE cohort are shown in detail in Table 6. Data of the 121 males with CAH are not shown. Missing values for sample characteristics were generally below 10%, with no missing data regarding age or gender and moderate missing frequencies (>10%) for working hours per week (290/1040, 27.9%) and feelings about household income (164/1040, 15.8%).

Table 6 Characteristics of the study sample (overall and in all dsd-LIFE countries)

The mean age of the participants was 32.4 years (SD 13.6), ranging from 25.2 (SD 10.1) in Poland to 39.4 (SD 14.7) in the Netherlands. The youngest participant was 16 years old, and the oldest was 75 years old. Compared with the general population (ESS) in their respective countries, the participants recruited in dsd-LIFE were 10–20 years “younger”, with the largest gap observed in Poland (mean age of the Polish population in ESS 46.1, SD 18.9).

In the country-specific analysis, the overall majority of participants were living as females (ranging from 59.8%, 73/122 in Sweden to 84.3%, 231/274 in France), except for in Poland (42.1%, 45/107), which was easily explained by varying recruitment numbers in the different diagnosis groups (Table 6). Regarding citizenship, the clear majority (more than 92%) hold citizenship in the country in which they live. This finding is very similar to the results from the ESS. Participants of dsd-LIFE were mainly living in large or small cities (ranging from approx. 45.3% in Sweden to 72.0% in Poland). Compared to ESS reference data, people living in large cities and suburbs were overrepresented in the dsd-LIFE population. Moreover, 6.4% of the participants had a migration background (first language spoken at home was not the same as the official language in the respective country).

Living circumstances varied among the countries (Table 6). The total “number of people living in the household” was similar across the study countries, with a mean between approximately two and three persons per household, and comparable to that of the general population (ESS).

In France and Poland, most of the participants were living with their parents (48.2% and 62.3%); in Sweden, the largest group of participants lived alone (37.0%); in Germany and the United Kingdom, the participants equally lived alone (30.3% and 28.6%), were married (18.5% and 23.8%) or lived with parents (30.3% and 28.6%), whereas in the Netherlands, most of the participants were married (46.1%).

Regarding education assessed via the ESS version of the International Standard Classification of EducationFootnote 1 (ESISCED; ranging from 1: basic education to 7: academic degree ≥ master), 20.5% of the DSD cohort from the dsd-LIFE cohort had a low education level (ESISCED ≤2), 46.1% had an intermediate education level (ESISCED 3–5), and 27.0% had high education levels, while the remaining 6.4% stated other education (e.g., school for children with learning difficulties). In general, the dsd-LIFE cohort had somewhat higher education levels than the ESS cohort with some differences across countries. In Germany, the distribution of categories was comparable to the ESS population, while in France, the Netherlands and more prominently in Sweden and the UK, we found a pattern of higher levels of education compared to the ESS cohort. Poland was the only example of somewhat lower educated participants compared to the general population.

Most subjects in the dsd-LIFE cohort performed paid work (52.9%) or had spent time in educational pursuits (16.1%) in the last 7 days. The remaining participants were either unemployed (8.3%), permanently sick or disabled (5.9%) or otherwise occupied in the last 7 days (retired, in community or military service, doing household work or other: 16.8%). This distribution was almost the same in all dsd-LIFE countries. Compared to the percentage reported in the ESS, the percentage of persons in paid work was very similar (e.g., Poland: 47.9% in dsd-LIFE vs. 48.3% in ESS), and the number of working hours per week was comparable among the dsd-LIFE countries (approx. 32–37 h, mean 33.6, SD 12.7). However, compared to those reported in the ESS, the number of working hours per week was 4 to 8 h less. In the small sample from the UK, there were more participants with full-time work and working hours comparable to the English ESS data, indicating some recruitment bias. The number of persons currently receiving education was two to three times higher in all countries, except in Poland and Sweden, than in the ESS (e.g., France: 18.4% in dsd-LIFE vs. 6.2% in ESS), which was explained by the lower mean age of the study population. In contrast, the proportion of retired people was much smaller in the dsd-LIFE cohort (e.g., France: 0.4% in dsd-LIFE vs. 32.0% in ESS).

Regarding household income, most of the dsd-LIFE participants were living comfortably or coping on their present income (both categories combined: 83.6%). This result was found for all dsd-LIFE countries (≥ 75% for combined categories). The results were also very similar to the country-specific outcomes of the ESS. The participants’ professions varied, but no ESS data for comparison were available. A detailed description is presented in Table 6.

In total, 17.8% of the dsd-LIFE cohort had contact with support groups in the last 12 months, varying between 4.4% in France and 35.3% in the Netherlands. Of these, 42–84% had contact via the Internet at least a few times, and 42–89% had contact in face-to-face meetings at least a few times.

Assessment of data quality

A total of 47 out of 1040 DSD participants (4.5%) did not respond to the patient-reported outcome questionnaires. The non-completion rate ranged from 0% (0/107) in Poland to 7.6% (19/231) in the Netherlands. A detailed overview of total non-response to the standardized instruments is shown in Table 7. The percentage of non-response to any entire questionnaire was 5–6% for the WHO-QOL-Bref, HADS, Adult ADHD Self-Report Scale (ASRS), Autism Spectrum Quotient (AQ10), RSES, and CSQ4. For the UGDS, the non-response rate for the entire questionnaire was 24% for the female version and 47% for the male version. Because several participants completed both female and male versions of this questionnaire, these percentages are based on all 1040 participants with DSD in the cohort. The Experiences in Close Relationships - Revised (ECR-RS) was not completed by 1.1% of subjects.Footnote 2 The frequency of missing items and missing sub-scores in completed questionnaires was less than 1.5% except for the UGDS scales, where 24.1% (female version) and 38.5% (male version) of the items were missing from the completed questionnaires (data not shown). Medical history was mandatory and was supplied by 99.5% of the participants. Very few participants did not meet in person with the study nurse or the physician but were willing to answer the online PRO at home. These participants were included in the study because diagnoses and data on medical history could be obtained from the medical records.

Table 7 Frequencies of missing questionnaires, sub-scores and items of standardized instruments according to diagnosis groups

A total of 928 (89.2%) of the 1040 DSD participants underwent at least part of the physical examination. Participation in the general examination by diagnosis group, country and gender is shown in Table 8. No relevant differences in the subgroups were found except for a lower participation rate in Sweden (65.6%, 80/122), whereas in Poland and the United Kingdom, all participants had at least a partial general examination. Participation in the gynaecological and urological examination is also shown in Table 8. Approximately one-half (346/717, 48.3%) of the females and 71.1% (221/311) of the males took part in the respective examinations. Across the countries, there was large variation in participation in gynaecological and urological examinations, reaching from none in the UK to almost all participants in Poland (both males and females). Participation in the examinations across diagnosis groups was more evenly distributed around the respective mean participation rate with the exception of the more frequent gynaecological examination in XY DSD females (92/142, 64.8%). In the group of XX CAH males, only 1 out of 5 took part in the urological examination. Metabolic parameters for 886 persons (85.2%) and hormones for 772 participants (74.2%) were determined. Ultrasound was performed on 380 (36.5%) of all 1040 DSD participants,Footnote 3 a spermiogramme was evaluated for 50 participants, a Dexa scan was carried out on 613 (58.9%) participants, and a BIA was performed on 283 participants (27.2%). Carotid IMT measurements were taken on 208 participants (20.0%).

Table 8 Participation in medical exams: general physical examination, gynaecological exam, urological exam per diagnosis group, country and gender

Retrospective data regarding the specific condition and treatment were available for 480/1040 (46.2%) of the participants. For the Turner group (152/301, 50.5%), the XY DSD group 108/222, 48.6%), the CAH group (131/226, 57.9%) and the smaller groups (including 47,XYY, XX DSD, 46, XX, and 45,X/46,XY + other Y: 46/73, 63.0%), the availability of at least some retrospective data was approximately the same. For the Klinefelter group, the availability of data was considerably lower (43/218, 19.7%). Gonadal histology was available for 123/1040 participants (11.8%), and for 500/1040 subjects (48.1%), self-reported information or data from medical reports were available regarding former condition-specific surgeries.

Discussion

The study sample

The dsd-LIFE consortium was able to recruit a large comprehensive DSD study population with 1040 participants. The initial sample size calculation had anticipated 1500 participants, but the size of the subgroups of the 1040 participants are large enough for adequate statistical analysis. It was easiest to recruit female participants with Turner syndrome and congenital adrenal hyperplasia (CAH) and men with Klinefelter syndrome and sample size was sufficient in these groups. The number of recruited participants with these specific diagnoses is consistent with the incidence of the diagnoses. Most patients were recruited with Turner syndrome (1:2500) followed by Klinefelter (1:500–1000), CAH (1:15,000) and XY DSD conditions (1:25–150,000). The group of XY DSD cases included the least common diagnoses that were clinically and/or genetically sub-classified. A total of 222 participated with only one third living in male gender (n = 73), covering a wide range of clinical diagnoses. Despite prolongation of the recruitment period for this group, it was most difficult to engage participants with XY-DSD conditions except for the centres in France, Poland and the Netherlands, which had specialized clinics for this group. Sufficient numbers for statistical analysis were achieved for the diagnoses of complete androgen insensitivity (CAIS), partial androgen insensitivity (PAIS), XY gonadal dysgenesis and 45,X/46,XY conditions but not for XY androgen synthesis defects. We conclude that despite the labelling of the study as dsd-LIFE, the group most closely associated with the term “disorders of sex development” was the most difficult to recruit. Adults with the condition appear to have less access to specialized centres or clinics, and appropriate treatment appears to be least efficient in this group [46]. We speculate that less regular contact and more variation in health care among this group may also contribute to some alienation. Equally, we were unable to recruit a sufficient number of participants with XX GD. We used mixed ways of recruitment via the medical centers, website, support groups with the aim to reach as many as possible individuals with rare diagnoses. Moreover, analysis of contact/membership of a support group as an influencing factor on outcomes is planned. It could be shown that the most efficient means of recruitment were direct contact and information provision about the study in the clinics, leading to a participation rate of 76%. The least successful means of recruitment was contact by mail, through support groups and website, with a participation rate of 22%, 24% and 0% respectively.

All participants could be classified according to the Chicago classification. However, for some participants with very unusual clinical presentation or unusual chromosomal or genetic findings, correct classification became a challenge. This was relevant, for example, in the 45,X/46,XY subgroup presenting with a non-androgenized Turner phenotype and individuals with androgenization. The latter group clinically resembles the XY DSD conditions and could be assigned to this group for most analyses. Similarly, the classification of 46,XX males and 47,XYY males proved to be difficult: clinically, these individuals are similar to Klinefelter males because they all display testicular dysgenesis and could be included in this group for outcome analyses. Moreover, the 46,XX gonadal dysgenesis patients included in the XX DSD subgroup are very different from 46,XX females with androgen effects due to CAH and should be considered as a separate group. Finally, 46,XX individuals with 21-OHD or 11ß-OHD deficiency living as males may be included or excluded in outcome analyses depending on the specific research question. We concede that the classification system from the Chicago Consensus Conference was very helpful in terms of a conceptual frame of reference. However, individual cases may still be difficult to allocate to one of the major groups. Any classification must be adapted to the purpose at hand, be it research, patient care or public communication.

Participation in different parts of the study

Participation in medical history and retrospective chart review

Partial to complete data on contemporary and past medical history from personal encounters with the physicians were obtained for 99.5% of the participants. Retrospective medical data from the chart review addressing the time of diagnosis and past treatment were obtained for only 46.2% of the participants. In the centres focusing on adult care, charts from previous treatment in childhood were not available. Moreover, for older participants, previous charts were not available because they were destroyed after 30 years in accordance with hospital regulations.

Participation in medical examinations

Most of the dsd-LIFE participants did take part in the general examination, with no relevant differences among diagnosis groups, gender or country. The participation rate in gynaecological or urological examinations varied between 15 and 97.8% and 46.9–98.4%, respectively. The participation rate in these intimate exams was highest in the XY DSD group, which may be interpreted as an indication of these participants’ need for gynaecological/urological follow-up and counselling. The participation rate in the Dexa scans (58.9%) was high, demonstrating that bone health is an issue for many persons with DSD conditions. The participation rate was lowest for the additional exams, such as genital ultrasound (36.5%), BIA (27.2%), carotid IMT (20%) and spermiograms (4.8%).

Response rate of the PRO

The rate of non-response to the entire PRO part of the dsd-LIFE was very low (4.5%). Especially, the standardized instruments used in the study showed almost complete response rates. The only exception was the UGDS, which had a non-response rate of 24.3% for the female and 47.2% for the male version. It is not entirely clear why questions regarding gender dysphoria were so often left unanswered. The UGDS was originally designed for transgender individuals without DSD but has been used without problems for people with DSD as well [47]. The low rate of responses to this particular instrument might indicate that there were misunderstandings or confusion regarding the items among our participants. Originally, the scale was presented in two gender-specific versions for those who live as female gender and those who live as male gender. In contrast to this presentation, i.e., in the German network study, the dsd-LIFE consortium wanted to allow people with non-binary identities (not identifying as either male or female) [48] to respond to items of both the male and female versions. This, however, may have resulted in people considering many questions as not applicable. For instance, persons identifying as females—most of our sample—may have omitted answers to all questions for males, but, being in a non-answering mode (the first half of the questionnaire considered questions for males), they may have omitted answers to questions for females as well.

We conclude that in general, most participants showed a very positive attitude toward answering the questionnaires and sharing their views on QoL, psychological well-being, treatment experiences, sexuality and ethical considerations.

Homogeneity/heterogeneity of countries

The characteristics that were heterogeneous among the dsd-LIFE countries were age, gender and diagnosis. Unequal distributions of diagnoses, age groups and gender per country may be explained by varied focuses of clinical care, research interests or access of dsd-LIFE study personnel to clinics in the centres. Therefore, the condition groups were not evenly distributed among all centres; all analysis concerning country differences must be controlled for the effects of the diagnosis, and vice versa.

Representativeness and limitations

The study sample is a convenience sample recruited by the specialized study centres with different clinical foci and in a variable proportion per country via support groups. A participation rate of 36% of all eligible persons in each centre cautions against generalizability. The participation rate ranged from 30.0% in the Netherlands to 54.4% in Sweden. Reasons for these differences are most likely explained by different approaches towards recruitment. Compared to the ESS sample, the sample for the current study was younger; the mean age was 32.36 years (SD 13.57) compared to the mean age of the ESS, which ranged from 48.70 years (SD 18.57) in Germany to 51.83 years (SD 19.12) in the United Kingdom. These age differences might be explained by the fact, that only in the recent years specialized clinics for adults with DSD conditions have been installed in several European countries. Patients are refered there mainly through transition programmes. Subsequently, few older patients are followed in these clinics. Most participants in the current analysis lived in an urban environment (large city to town), had a slightly higher education level and were satisfied or coping well on their present income. The sample size of females was approximately double that of males. Only a few participants reported to have a gender other than male or female; those participants were recruited in Germany and the Netherlands. In those two countries, there is—perhaps more than in the other countries—discussion in support groups and among the general public on the acknowledgement of non-binary identities. For instance, in a Dutch study on the prevalence of gender dysphoria, 3–4% reported to have an ambivalent gender identity (equal identification with the male and female genders) [49]. However, a selection bias such as that in our study is well known from previous studies in clinical samples. This study has attempted to document at least the number of potential participants from hospital data identifying potential participants, who were all approached and informed. Through this process, numbers of contacted persons, ways of contact and participation rate could be retrieved. Unfortunately, specific information on the diagnoses of non-participants and their reasons not to participate could not be obtained because the ethical restrictions did not allow any data on non-participants to be collected. We are therefore unable to provide a non-responder analysis.

Data quality

Overall, the data quality of the study is good because the mandatory parts of the study, such as medical history and PRO, showed very low rates of completely missing data of 0.5% and 4.5%, respectively. Subsequently, for most analyses of the cohort, there were ample data for statistical analysis. In addition, for most rare diagnoses, such as 45,X/46,XY, CAIS, PAIS, partial XY gonadal dysgenesis and CAH, data are estimated to be adequate for clinical recommendations. Only for some very rare diagnoses, such as XY complete gonadal dysgenesis, XX gonadal dysgenesis and androgen synthesis defects, were the numbers of participants and data insufficient for statistical analysis. However, these rare diagnoses are of special interest and will be described as case series.

Analyses and publication strategy

The main data analyses for the primary objectives will be performed by the coordination centre for clinical studies of the Charité Universitätsmedizin Berlin. Secondary analyses addressing specific areas of interest or related to clinical subgroups will be performed by author teams composed of members of the dsd-LIFE consortium based on a statistical analysis plan and a written publication proposal. In addition, the results will be presented to the dsd-LIFE recommendation groups that augment the existing knowledge on health care for these conditions with new information from the study. Dissemination strategies will include scientific publications, presentations within the scientific community and associations of health care professionals, support groups, and other stakeholders and policymakers.

Conclusions

dsd-LIFE is the first and largest European cross-sectional study including the majority of conditions encompassed by the DSD classification. The data analyses will focus on issues that are important for the improvement of care and the development of clinical recommendations for patients with these conditions, such as hormone therapy, surgery, fertility, psychological and social support, and psychosexual and ethical issues. Moreover, the specific aspects of single conditions, such as Turner syndrome and Klinefelter syndrome, 45,X/46,XY, CAIS, PAIS and CAH will be considered for improvement of care. A special focus will be to analyse and consider the participants’ views on the different issues. Altogether, QoL, psychosexual issues, physical and mental health and satisfaction with treatment, support and helath services will be measured in the whole cohort and in sub-samples if indicated. The data from this large sample will provide a sufficient basis for evidence-based recommendations for improvement of clinical care of individuals affected by a DSD condition.

Notes

  1. ESISCED categories: 1: less than lower secondary; 2: lower secondary; 3: lower tier upper secondary; 4: upper tier upper secondary; 5: advanced vocational, sub-degree; 6: lower tertiary education, BA level; 7: higher tertiary education, ≥ MA level

  2. The ECR-RS could be answered only by participants who answered ―yes to the question: ―Have you ever had a “romantic” relationship? A total of 700 participants (67.3%) answered yes, while 273 (26.3%) answered ―no and 67 (6.4%) did not answer the question.

  3. In France, one participant received both ultrasound examinations.

Abbreviations

11ß-OHD:

11ß-hydroxylase deficiency

21-OHD:

21-OH deficiency: 21-hydroxylase deficiency

46,XX DSD:

46,XX disorders/differences of sex development

46,XY DSD:

46,XY disorders/differences of sex development

AMH:

Anti-muellerian hormone

AQ10:

Autism Spectrum Quotient (AQ)

ASRS:

Adult ADHD Self-Report Scale

BIA:

Bioelectrical impedance analysis

CAH:

Congenital adrenal hyperplasia

CAIS:

Complete androgen insensitivity

CHC-SUN:

Child Health Care – Satisfaction, Utilization, Needs Questionnaire

CRF:

Case report form

CSQ4:

Client Satisfaction Questionnaire

DEXA:

Dual-energy X-ray absorptiometry

DSD:

Disorders/differences of sex development

ECR-RS:

Experiences in Close Relationships - Revised

ESISCED:

European Social Survey version of the International Standard Classification of Education

ESS:

European Social Survey

HADS:

Hospital anxiety and depression scale

HRQOL:

Health-related quality of life

I-DSD:

International DSD database

IMT:

Intima media thickness

KKS:

Koordinierungszentrum für klinische Studien

PAIS:

Partial androgen insensitivity

PRO:

Patient-reported outcome

QoL:

Quality of life

R:

Software environment for statistical computing

SAS:

Statistical analysis software

SOP:

Standard operational procedures

STC:

Steering committee

UGDS:

Utrecht Gender Dysphoria Scale

WHO-QOL-Bref:

World Health Organization Quality of Life-Bref is a shorter version of the original instrument to assess Quality of Life

References

  1. Lee PA, Houk CP, Ahmed SF, Hughes IA. Society icwtpitICCoIobtLWPE, endocrinology tESfP: consensus statement on Management of Intersex Disorders. Pediatrics. 2006;118:e488–500.

    Article  PubMed  Google Scholar 

  2. Lee PA, Nordenstrom A, Houk CP, Ahmed SF, Auchus R, Baratz A, Baratz Dalke K, Liao LM, Lin-Su K, Looijenga LH 3rd, et al. Global disorders of sex development update since 2006: perceptions, approach and care. Horm Res Paediatr. 2016;85:158–80.

    Article  CAS  PubMed  Google Scholar 

  3. Sandberg D, Mazur T: A Noncategorial Approach to the Psychosocial Care of Persons with DSD and Their Famlilies. In Gender Dysphoria and Disorders of Sex Development. Volume 1. Edited by Kreukels B, Steensma T, de Vries A. New York: Springer Science+Business Media. 2014;2014:93–114.

  4. Levitsky LL, Luria AH, Hayes FJ, Lin AE. Turner syndrome: update on biology and management across the life span. Curr Opin Endocrinol Diabetes Obes. 2015;22:65–72.

    Article  CAS  PubMed  Google Scholar 

  5. Bondy CA. Care of girls and women with turner syndrome: a guideline of the turner syndrome study group. J Clin Endocrinol Metab. 2007;92:10–25.

    Article  CAS  PubMed  Google Scholar 

  6. Nieschlag E. Klinefelter syndrome: the commonest form of hypogonadism, but often overlooked or untreated. Dtsch Arztebl Int. 2013;110:347–53.

    PubMed  PubMed Central  Google Scholar 

  7. de Neve-Enthoven NG, Callens N, van Kuyk M, van Kuppenveld JH, Drop SL, Cohen-Kettenis PT, Dessens AB. Psychosocial well-being in Dutch adults with disorders of sex development. J Psychosom Res. 2016;83:57–64.

    Article  PubMed  Google Scholar 

  8. D'Alberton F, Assante MT, Foresti M, Balsamo A, Bertelloni S, Dati E, Nardi L, Bacchi ML, Mazzanti L. Quality of life and psychological adjustment of women living with 46,XY differences of sex development. J Sex Med. 2015;12:1440–9.

    Article  PubMed  Google Scholar 

  9. Amaral RC, Inacio M, Brito VN, Bachega TA, Oliveira AA, Jr., Domenice S, Denes FT, Sircili MH, Arnhold IJ, Madureira G, et al: Quality of life in a large cohort of adult Brazilian patients with 46,XX and 46,XY disorders of sex development from a single tertiary centre. Clin Endocrinol (Oxf). 2015;82(2):274–9.

  10. Köhler B, Kleinemeier E, Lux A, Hiort O, Gruters A, Thyen U. Grp DSDNW: satisfaction with genital surgery and sexual life of adults with XY disorders of sex development: results from the German clinical evaluation study. Journal of Clinical Endocrinology & Metabolism. 2012;97:577–88.

    Article  Google Scholar 

  11. Reisch N, Arlt W, Krone N. Health problems in congenital adrenal hyperplasia due to 21-Hydroxylase deficiency. Hormone Research in Paediatrics. 2011;76:73–85.

    Article  CAS  PubMed  Google Scholar 

  12. Arlt W, Willis DS, Wild SH, Krone N, Doherty EJ, Hahner S, Han TS, Carroll PV, Conway GS, Rees DA, et al. Health status of adults with congenital adrenal hyperplasia: a cohort study of 203 patients. J Clin Endocrinol Metab. 2010;95:5110–21.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Schützmann K, Brinkmann L, Schacht M, Richter-Appelt H. Psychological distress, self-harming behavior, and suicidal tendencies in adults with disorders of sex development. Arch Sex Behav. 2009;38:16–33.

    Article  PubMed  Google Scholar 

  14. Johannsen TH, Ripa CPL, Mortensen EL, Main KM. Quality of life in 70 women with disorders of sex development. Eur J Endocrinol. 2006;155:877–85.

    Article  CAS  PubMed  Google Scholar 

  15. Migeon CJ, Wisniewski AB, Gearhart JP, Meyer-Bahlburg HFL, Rock JA, Brown TR, Casella SJ, Maret A, Ngai KM, Money J, Berkovitz GD. Ambiguous genitalia with perineoscrotal hypospadias in 46,XY individuals: long-term medical, surgical, and psychosexual outcome. Pediatrics. 2002;110:10.

    Article  Google Scholar 

  16. Groth KA, Skakkebaek A, Host C, Gravholt CH, Bojesen A. Clinical review: Klinefelter syndrome--a clinical update. J Clin Endocrinol Metab. 2013;98:20–30.

    Article  CAS  PubMed  Google Scholar 

  17. Herlihy AS, McLachlan RI, Gillam L, Cock ML, Collins V, Halliday JL. The psychosocial impact of Klinefelter syndrome and factors influencing quality of life. Genet Med. 2011;13:632–42.

    Article  PubMed  Google Scholar 

  18. Fjermestad KW, Naess EE, Bahr D, Gravholt CH. A 6-year follow-up survey of health status in middle-aged women with turner syndrome. Clin Endocrinol. 2016;85:423–9.

    Article  Google Scholar 

  19. Naess EE, Bahr D, Gravholt CH. Health status in women with turner syndrome: a questionnaire study on health status, education, work participation and aspects of sexual functioning. Clin Endocrinol. 2010;72:678–84.

    Article  Google Scholar 

  20. Lux A, Kropf S, Kleinemeier E, Jürgensen M, Thyen U. Grp DSDNW: clinical evaluation study of the German network of disorders of sex development (DSD)/intersexuality: study design, description of the study population, and data quality. BMC Public Health. 2009;9

  21. Colombetti E, Osimani B, Aluas M, Pessina A, Musio A. Revision of international classification of functioning, disability and health ethical guidelines international classification of functioning, disability and health-related ethical issues. American Journal of Physical Medicine & Rehabilitation. 2012;91:S155–8.

    Article  Google Scholar 

  22. Bickenbach J. Ethics, disability and the international classification of functioning, disability and health. American Journal of Physical Medicine & Rehabilitation. 2012;91:S163–7.

    Article  Google Scholar 

  23. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25:3186–91.

    Article  CAS  PubMed  Google Scholar 

  24. Acquadro C, Jambon B, Ellis D, Marquis P. Language and translation issues. Quality of life and Pharmaco-economics in clinical trials. Philadelphia: Lippincon-Raven Publishers.

  25. Harper A, Power M, Grp W. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med. 1998;28:551–8.

    Article  Google Scholar 

  26. Herrman H, Metelko Z, Szabo S, Rajkumar S, Kumar S, Vanheck G, Devries J, Sucre A, Kabanov, Lomachenkov, et al. Study protocol for the word health organization project to develop a quality-of-life assessment instrument (WHOQOL). Qual Life Res. 1993;2:153–9.

    Article  Google Scholar 

  27. Bjelland I, Dahl AA, Haug TT, Neckelmann D. The validity of the hospital anxiety and depression scale - an updated literature review. J Psychosom Res. 2002;52:69–77.

    Article  PubMed  Google Scholar 

  28. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67:361–70.

    Article  CAS  PubMed  Google Scholar 

  29. Lindgren TW, Pauly IB. Body image scale for evaluating transsexuals. Arch Sex Behav. 1975;4:639–56.

    Article  CAS  PubMed  Google Scholar 

  30. van de Grift TC, Cohen-Kettenis PT, Steensma TD, De Cuypere G, Richter-Appelt H, Haraldsen IRH, Dikmans REG, Cerwenka SC, Kreukels BPC. Body satisfaction and physical appearance in gender Dysphoria. Arch Sex Behav. 2016;45:575–85.

    Article  PubMed  Google Scholar 

  31. CohenKettenis PT. vanGoozen SHM: sex reassignment of adolescent transsexuals: a follow-up study. J Am Acad Child Adolesc Psychiatry. 1997;36:263–71.

    Article  CAS  Google Scholar 

  32. Schonbucher V, Schweizer K, Rustige L, Schutzmann K, Brunner F, Richter-Appelt H. Sexual quality of life of individuals with 46,XY disorders of sex development. Journal of Sexual Medicine. 2012;9:3154–70.

    Article  PubMed  Google Scholar 

  33. Kinsey AC, Pomeroy WB, Martin CE, Gebbard PH. Sexual behavior in the human female. Philadelphia, London: W.B. Saunders Company; 1953.

    Google Scholar 

  34. Kinsey AC, Pomeroy WB, Martin CE. Sexual behavior in the human male. Bloomington: Indiana U. Press; 1948.

    Google Scholar 

  35. Schmidt S, Thyen U, Chaplin J, Mueller-Godeffroy E, European DG. Cross-cultural development of a child health care questionnaire on satisfaction, utilization, and needs. Ambul Pediatr. 2007;7:374–82.

    Article  PubMed  Google Scholar 

  36. Schmidt S, Thyen U, Herrmann-Garitz C, Bomba F, Muehlan H. The youth health care measure-satisfaction, utilization, and needs (YHC-SUN)-development of a self-report version of the child health care (CHC-SUN) proxy-measure. BMC Health Serv Res. 2016;16:12.

    Article  Google Scholar 

  37. DeutscherEthikrat (Ed.). Intersexualität - Stellungsnahme, Vorsitzender: Prof. Dr. Edzard Schmidt-Jortzig edition. http://www.ethikrat.org/publikationen/stellungnahmen/intersexualitaet (23.01.2015); 2012.

  38. Wiesemann C, Ude-Koeller S, Sinnecker GH, Thyen U. Ethical principles and recommendations for the medical management of differences of sex development (DSD)/intersex in children and adolescents. Eur J Pediatr. 2010;169:671–9.

    Article  PubMed  Google Scholar 

  39. use ICoHotrfropfh: Harmonised tripartite guideline - clinical Invenstigation of medicinal products in the pediatric population - E11. 2000, Step 4 version.

    Google Scholar 

  40. Association WM: WMA declaration of Helsinki - ethical principles for medical research involving human subjects. 2008.

    Google Scholar 

  41. Europe Co: Convention for the protection of human rights and dignity of the human being with regard to the application of biology and medicine: convention on human rights and biomedicine. 1997, ETS 164 - 4.IV.1997.

    Google Scholar 

  42. European Social Survey. Source Questionnaire Amendment 01 (Round 6, 2012/2013); 2012. p. 1–67.

    Google Scholar 

  43. Mortensen KH, Andersen NH, Gravholt CH. Cardiovascular phenotype in turner syndrome--integrating cardiology, genetics, and endocrinology. Endocr Rev. 2012;33:677–714.

    Article  CAS  PubMed  Google Scholar 

  44. Wedell A, Ritzen EM, Haglundstengler B, Luthman H. Steroid 21-hydroxylase deficiency-3 additional mutated alleles and establishment of phenotype genotype relationships of common mutations. Proc Natl Acad Sci U S A. 1992;89:7232–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Krone N, Rose IT, Willis DS, Hodson J, Wild SH, Doherty EJ, Hahner S, Parajes S, Stimson RH, Han TS, et al. Genotype-phenotype correlation in 153 adult patients with congenital adrenal hyperplasia due to 21-Hydroxylase deficiency: analysis of the United Kingdom congenital adrenal hyperplasia adult study executive (CaHASE) cohort. Journal of Clinical Endocrinology & Metabolism. 2013;98:E346–54.

    Article  CAS  Google Scholar 

  46. Thyen U, Lux A, Jurgensen M, Hiort O, Kohler B. Utilization of health care services and satisfaction with Care in Adults Affected by disorders of sex development (DSD). J Gen Intern Med. 2014;29(Suppl 3):752–9.

    Article  PubMed Central  Google Scholar 

  47. Jurgensen M, Kleinemeier E, Lux A, Steensma TD, Cohen-Kettenis PT, Hiort O, Thyen U, Kohler B. Grp DSDNW: psychosexual development in adolescents and adults with disorders of sex development-results from the German clinical evaluation study. Journal of Sexual Medicine. 2013;10:2703–14.

    Article  PubMed  Google Scholar 

  48. Richards C, Bouman WP, Seal L, Barker MJ, Nieder TO, T'Sjoen G. Non-binary or genderqueer genders. Int Rev Psychiatry. 2016;28:95–102.

    Article  PubMed  Google Scholar 

  49. Kuyper L, Wijsen C. Gender identities and gender Dysphoria in the Netherlands. Arch Sex Behav. 2014;43:377–85.

    Article  PubMed  Google Scholar 

  50. Amaral RC, Inacio M, Brito VN, Bachega TA, Domenice S, Arnhold IJ, Madureira G, Gomes L, Costa EM, Mendonca BB: Quality of life of patients with 46,XX and 46,XY disorders of sex development. Clin Endocrinol (Oxf) 2014.

  51. Kessler RC, Adler L, Ames M, Demler O, Faraone S, Hiripi E, Howes MJ, Jin R, Secnik K, Spencer T, et al. The World Health Organization adult ADHD self-report scale (ASRS): a short screening scale for use in the general population. Psychol Med. 2005;35:245–56.

    Article  PubMed  Google Scholar 

  52. Adler LA, Shaw DM, Spencer TJ, Newcorn JH, Hammerness P, Sitt DJ, Minerly C, Davidow JV, Faraone SV. Preliminary examination of the reliability and concurrent validity of the attention-deficit/hyperactivity disorder self-report scale v1.1 symptom checklist to rate symptoms of attention-deficit/hyperactivity disorder in adolescents. Journal of Child and Adolescent Psychopharmacology. 2012;22:238–44.

    Article  PubMed  Google Scholar 

  53. Sonnby K, Skordas K, Olofsdotter S, Vadlin S, Nilsson KW, Ramklint M. Validation of the World Health Organization adult ADHD self-report scale for adolescents. Nordic Journal of Psychiatry. 2015;69:216–23.

    Article  PubMed  Google Scholar 

  54. Allison C, Auyeung B, Baron-Cohen S. Toward brief "red flags" for autism screening: the short autism Spectrum quotient and the short quantitative checklist in 1,000 cases and 3,000 controls. J Am Acad Child Adolesc Psychiatry. 2012;51:202–12.

    Article  PubMed  Google Scholar 

  55. Rosenberg M, Schooler C, Schoenbach C, Rosenberg F. GLOBAL SELF-ESTEEM AND SPECIFIC SELF-ESTEEM - DIFFERENT CONCEPTS, DIFFERENT OUTCOMES. Am Sociol Rev. 1995;60:141–56.

    Article  Google Scholar 

  56. Franck E, De Raedt R, Barbez C, Rosseel Y. PSYCHOMETRIC PROPERTIES OF THE DUTCH ROSENBERG SELF-ESTEEM SCALE. Psychologica Belgica. 2008;48:25–34.

    Article  Google Scholar 

  57. Kleinemeier E, Jurgensen M, Lux A, Widenka PM, Thyen U. Grp DSDNW: psychological adjustment and sexual development of adolescents with disorders of sex development. J Adolesc Health. 2010;47:463–71.

    Article  PubMed  Google Scholar 

  58. Fraley RC, Heffernan ME, Vicary AM, Brumbaugh CC. The experiences in close relationships-relationship structures questionnaire: a method for assessing attachment orientations across relationships. Psychol Assess. 2011;23:615–25.

    Article  PubMed  Google Scholar 

  59. Donbaek DF, Elklit A. A validation of the experiences in close relationships-relationship structures scale (ECR-RS) in adolescents. Attach Hum Dev. 2013;

  60. Kreukels BPC, Haraldsen IR, De Cuypere G, Richter-Appelt H, Gijs L, Cohen-Kettenis PT. A European network for the investigation of gender incongruence: the ENIGI initiative. European Psychiatry. 2012;27:445–50.

    Article  CAS  PubMed  Google Scholar 

  61. Schneider C, Cerwenka S, Nieder TO, Briken P, Cohen-Kettenis PT, De Cuypere G, Haraldsen IR, Kreukels BPC, Richter-Appelt H. Measuring gender Dysphoria: a multicenter examination and comparison of the Utrecht gender Dysphoria scale and the gender identity/gender Dysphoria questionnaire for adolescents and adults. Arch Sex Behav. 2016;45:551–8.

    Article  PubMed  Google Scholar 

  62. Attkisson CC, Zwick R, et al. Evaluation and program planning. 1982;5:233–7.

    Article  CAS  PubMed  Google Scholar 

  63. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Evaluation and program planning. 1979;2:197–207.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We are grateful to the persons who participated in dsd-LIFE and to all of the study centres for their enthusiasm and dedication to contacting potential participants and collecting high-quality data. We thank the following people at each dsd-LIFE study centre:

Berlin: Elena Bennecke (data collection, psychological support), Elena Ioana Braicu and Nicole Gehrmann (gynaecological examination and counselling), The-Hoang Do (construction of online database, data management), Felix Frömel and Maria Wiese (construction of online database), Katharina Gehrmann (scientific project management), Alexander Krannich, Robert Röhle and Gerald Splettstößer (biostatistical support), Anja Lingnau (urological examination and counselling), Katja Niyet (study nurse), Ursula Plöckinger and Gertraud Schmid (body composition measurement), Marcus Quinkler (recruitment, design of medical questionnaire), Karin Rothe (surgical counselling), Jalid Sehouli (gynaecological examination and counselling), Knut Werner-Rosen (recruitment and psychological support), Erwin Lankes (medical examination), Uta Neumann (medical examination). Lübeck: Wiebke Birnbaum (data collection), Frau Dmoch (study nurse), Lou Marshall (psychosocial support, data collection), Axa Marxsen (data collection and medical examination), Esther Müller-Godeffroy (design and methodology of the study), Marion Rapp (scientific project management), Kianush Tafazzoli and Lutz Wünsch (urological examination), Karen Wimmer (data collection and gynaecological examination). Münster: Nadine Grossmüller-Sielenkemper (recruitment, medical examination), Susanne Krege (surgical counselling). Amsterdam: Tim van der Grift (Recruitment, data collection, data analysis), Martin den Heijer (Klinefelter recommendation group), Baudewijntje Kreukels (project coordinator, data management, psychological recommendation group, data analysis and interpretation). Nijmegen: Janielle van Alfen, Barbara Kortmann and Nike Stikkelbroeck (recruitment, data collection, analysis, interpretation), Viviane vd Crommert and Karen Kwak (study nurse), Wout Feitz and Ad Hermus (recruitment), Chris Verhaak (analysis, interpretation), Jos Gilissen (coordinator research centre in Nijmegen, conception of work, design of the Teun van Herwaarden (laboratory medicine, storage of samples) Kathleen d Houwers and Henri Timmers (recruitment, analysis, interpretation), Regine Keting (project management support), Loes Kooijman (data collection, project management support). Birmingham: Elizabeth S. Baranowski, Christopher M. Jones and Emma A. Webb (recruitment, data collection and medical examination), Miriam Asia and Chona Feliciano (study nurse). Toulouse: Françoise Lorenzini, Sophie Bezio and Solange Grunenwald and Audrey Cartault–Grandmottet (data collection, recruitment, medical examination, gynaecological examination and publication participation), Béatrice Jouret, Philippe Galinier, Sofia Mouttalib (medical examination and publication participation), Adams Hillard, Thomas Edouard (data analysis and publication participation), Catherine Arnaud (epidemiologist, data analysis and publication participation), Francoise Auriol (coordinator of paediatric research centre, data collection), Isabelle Kieffer (clinical researcher in paediatric research centre, data collection),Sophie Mourgues (clinical researcher in the Clinical Research Department of CHU), Jean Pierre Salles and Maithé Tauber (publication participation), Florence Tremollières (bone densitometry, data collection). Montpellier: Sylvie Broussous (data management, recruitment), Isabelle Raingeard (medical examination). Lyon: Caroline Demilly (psychological support), Daniela Gorduzza (data collection and urological examination), Laurence Heitz (study nurse), Sandrine Masson (design and methodology of the study), Delphine Maucort-Boulch (design and methodology of the study and biostatistical support), André Serusclat (radiology), Yvonne Varillon (data collection). München: Heike Kuenzel (psychological support), Christian Lottspeich (data collection and examination), Kerstin Schaefer (data collection and examination). Stockholm: Stefan Arver and Mats Holmberg (recruitment, data collection, medical and urological examination), Liselott Blomberg, Susanne Jarlvik Alm and Carina Levelind (study nurse), Birgit Borgstroem (recruitment, data management), Urban Ekström (data collection, medical and urological examination), Hedvig Engberg (recruitment, data collection and management, medical and gynaecological examination), Henrik Falhammar (recruitment), Louise Frisén and Agneta Nordenskjöld (study design, recruitment), Svetlana Lajic (recruitment, data collection), Angelica Lindén Hirschberg (recruitment, gynaecological examination), Carl Lundeberg (data collection and management), Anna Strandqvist (data collection and management, psychological support). Lodz: Katarzyna Bajszczak (psychological support), Katarzyna Marchlewska (statistician), Elzbieta Oszukowska (urological examination and counselling), Renata Walczak-Jedrzejowska (preparation of clinical material). Göttingen: Stephanie Bernstein (design of study and instruments), Friederike Maaßen (data evaluation and interpretation). Warszawa: Katarzyna Bajszczak (psychological support), Agnieszka Lecka-Ambroziam and Agnieszka Rudzka-Kocjan (data collection), Mieczysław Szalecki (Head of Endocrinology Department). Paris: Anne Laure Castell (medical examination, recruitment), Lise Duranteau (concept of the study, data collection, data evaluation), Jennifer Flandrin (gynaecological examination, data collection), Anne Sophie Lambert (recruitment), Luigi Maione (Department of Endocrinology and Reproductive Diseases, biological data).

We especially thank the support groups in the different countries that have supported the study: Intersexuelle Menschen e.V., Turner-Syndrom-Vereinigung Deutschland e.V., 47xxy klinefelter syndrome e.v., AGS-Eltern- und Patienteninitiative e.V. (Germany); Nederlandse Klinefelter Vereniging, Dutch Adrenal Society, DSDNederland and Turner Contact Nederland (the Netherlands); and Association surrenales (France). Holger Muehlan from the Department of Health & Prevention at the Institute of Psychology of the University of Greifswald conducted the psychometric evaluation of the instrument for satisfaction with care (CHC-SUN) for the purposes of this study.

The study was supervised by the following Scientific Advisory Board: Prof. Peter Lee, Prof. Heino Meyer-Bahlburg, Prof. Inas Mazen, Prof. Justine Schober, Dr. Margaret Simmonds, and Dr. Jürg Streuli.

The dsd-LIFE study group

Birgit Köhler; Peggy Cohen-Kettenis; Annelou de Vries; Wiebke Arlt; Claudia Wiesemann; Jolanta Slowikowska-Hilczer; Aude Brac de la Perriere; Charles Sultan; Francoise Paris; Claire Bouvattier; Ute Thyen; Nicole Reisch; Annette Richter-Unruh; Hedi Claahsen-van der Grinten; Anna Nordenström; Catherine Pienkowski; Maria Szarras-Czapnik.

Funding

European Union Seventh Framework Programme (FP7/2007–2013) under grant agreement n°305373. http://www.dsd-life.eu/.

Availability of data and materials

The datasets generated and analysed during the current study are not publicly available because data analysis will mainly be performed by the dsd-LIFE consortium.

Author information

Authors and Affiliations

Authors

Consortia

Contributions

RR contributed in the conception of the article and carried out the statistical analysis and interpretation of data. KG contributed in the conception of the article and carried out the description of the instruments used. MSC, HCvdG, CP, CB, PCK and AN contributed in classification of the participants, interpretation of data and revision. UT and BK carried out the conception of the article, the interpretation of data and the final approval. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Birgit Köhler.

Ethics declarations

Ethics approval and consent to participate

We state that ethical approvals were obtained for performance of the study by each participating centre; see also chapter ethics/study design and methods. The ethics committee of the coordinator, which first approved the study, was the Ethics Commission of the Charité Universitätsmedizin, reference number EA2/069/13.

Consent for publication

Informed consent for publication of anonymised data was given by all participants; see also chapter ethics/study design and methods. If participants were underage, their parents gave their respective informed consent.

Competing interests

dsd-LIFE does not involve the interests of pharma. The authors declare that they do not have any conflicts of interest.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Röhle, R., Gehrmann, K., Szarras-Czapnik, M. et al. Participation of adults with disorders/differences of sex development (DSD) in the clinical study dsd-LIFE: design, methodology, recruitment, data quality and study population. BMC Endocr Disord 17, 52 (2017). https://0-doi-org.brum.beds.ac.uk/10.1186/s12902-017-0198-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12902-017-0198-y

Keywords