Figures
Abstract
Objective
Although metabolic profiles have been associated with chronic disease risk, lack of temporal stability of metabolite levels could limit their use in epidemiological investigations. The present study aims to evaluate the reliability over a two-year period of 158 metabolites and compare reliability over time in fasting and non-fasting serum samples.
Methods
Metabolites were measured with the AbsolueIDQp180 kit (Biocrates, Innsbruck, Austria) by mass spectrometry and included acylcarnitines, amino acids, biogenic amines, hexoses, phosphatidylcholines and sphingomyelins. Measurements were performed on repeat serum samples collected two years apart in 27 fasting men from Turin, Italy, and 39 non-fasting women from Utrecht, The Netherlands, all participating in the European Prospective Investigation into Cancer and Nutrition (EPIC) study. Reproducibility was assessed by estimating intraclass correlation coefficients (ICCs) in multivariable mixed models.
Results
In fasting samples, a median ICC of 0.70 was observed. ICC values were <0.50 for 48% of amino acids, 27% of acylcarnitines, 18% of lysophosphatidylcholines and 4% of phosphatidylcholines. In non-fasting samples, the median ICC was 0.54. ICC values were <0.50 for 71% of acylcarnitines, 48% of amino acids, 44% of biogenic amines, 36% of sphingomyelins, 34% of phosphatidylcholines and 33% of lysophosphatidylcholines. Overall, reproducibility was lower in non-fasting as compared to fasting samples, with a statistically significant difference for 19–36% of acylcarnitines, phosphatidylcholines and sphingomyelins.
Citation: Carayol M, Licaj I, Achaintre D, Sacerdote C, Vineis P, Key TJ, et al. (2015) Reliability of Serum Metabolites over a Two-Year Period: A Targeted Metabolomic Approach in Fasting and Non-Fasting Samples from EPIC. PLoS ONE 10(8): e0135437. https://doi.org/10.1371/journal.pone.0135437
Editor: Erica Villa, University of Modena & Reggio Emilia, ITALY
Received: March 4, 2015; Accepted: July 23, 2015; Published: August 14, 2015
Copyright: © 2015 Carayol et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: This study has been led by researchers from the International Agency for Research on Cancer (IARC, Lyon, France). Participants’ blood samples have initially been collected in EPIC centers from Turin (Italy) and Utrecht (The Netherlands). The study dataset has been centralized at IARC and statistical analyses carried out under the supervision of Dr. Pietro Ferrari. The primary responsibility for accessing the data belongs to Turin and Utrecht EPIC centers that provided the blood samples. Their use as anonymized dataset could be made available by contacting the corresponding author, Dr. Pietro Ferrari, ferrarip@iarc.fr, who will issue a request to scientists of EPIC Utrecht and Turin.
Funding: This work was supported by the European Union (EUROCAN FP7-KBBE-2010.2.4.1-2 grant #260791). AS received the funding. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The work reported in this paper was undertaken during the tenure of MC's postdoctoral fellowship awarded by the International Agency for Research on Cancer.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Advancement of analytical technologies has made high-throughput metabolic profiling of biological specimens possible [1]. Mass spectrometry techniques have been increasingly used to characterize the human metabolome [1,2] and the complex metabolic effects of nutrients or foods on chronic diseases in large-scale epidemiological studies [3–7], mainly because of their sensitivity and selectivity. However, most of the epidemiological studies rely on a single measurement of the metabolome. An eventual lack of temporal stability of the metabolome may bias relative risks based on a single measurement towards the null and could limit the significance of the findings [8–10].
Reliability studies evaluate the reproducibility of measurements taken on the same individuals at separate times. Reliability over time improves as the proportion of the total variance due to the between-subject variance increases. The reliability over time of metabolic profiles has recently been investigated in metabolites measured in blood samples [11–15]. Overall, fair to good reliability was reported, particularly in amino acids, hexoses, phosphatidylcholines, and sphingomyelins [11,12,15], except some acylcarnitines compounds that may yield poorer reliability [11,12]. Those reliability studies focused on assessing the temporal stability of metabolites measured in fasting blood samples. However, some epidemiological studies investigating metabolomics-disease links have used blood samples from non-fasting participants [2,16,17]. As food intake influences the metabolome [18], it has been hypothesized that the reliability was weaker in non-fasting than in fasting samples. This motivates the need of a study to compare the reliability of fasting and non-fasting samples.
The present study aims (1) to evaluate the reliability of metabolic profiling measurements over time in serum samples collected on average 2 years apart in subjects participating in the European Prospective Investigation into Cancer and Nutrition (EPIC) study, separately for fasting and non-fasting measures, and (2) to statistically compare reliability over time in fasting and non-fasting subgroups.
Materials and Methods
Study population and blood collection
EPIC is a large European cohort involving more than 521,000 subjects from 23 centers in 10 European countries [19]. All participants gave written informed consent to the EPIC study and the Ethical Committee of the International Agency of Research on Cancer specifically approved the present study. The EPIC cohort has at present an average follow-up of about 11 years. All participants completed questionnaires on diet, lifestyle, and medical history. In addition, for about 80% of EPIC participants a blood sample was collected at recruitment [19]. The EPIC cohort is particularly suitable to carry out epidemiological investigations on the link between metabolomics and risk of cancer. The current study was based on a sub-sample of participants who gave blood samples at two different time points: 39 non-fasting women from Utrecht (the Netherlands) and 27 fasting men from Turin (Italy). All the participants included in the present study were free of disease (diabetes, stroke, heart and overall cancer) at baseline. Although fasting and non-fasting individuals were different regarding gender and origin, this specific population was chosen as information about fasting conditions in these EPIC participants were known and consistent at both time points.
In Utrecht, women aged 50–65 years old, with no history of diabetes, cardiovascular disease or cancer, were invited to participate in a breast cancer screening from 1993 to 1999. A second examination with blood collection was carried out in a subsample of the cohort. Among all women attending this second examination, 39 who gave non-fasting blood samples in both occasions at 2.4 years interval (5th-95th: 2.2–3.9) were selected. In Turin, blood donors men aged 35–64 years were recruited from 1993 to 1998. A second blood drawn was taken in 39 men of which 27 who gave fasting blood samples in both occasions at 1.9 years interval (5th-95th: 1.0–2.8) were selected.
All the collected specimens were treated and banked with the same protocol as previously detailed for the whole cohort [20,21]. In short, blood samples were drawn in tubes without any anticoagulant (serum fraction), stored at +4°C until centrifugation (4,000 rpm for 20 minutes), and then stored in liquid nitrogen at -196°C.
Biomarker analyses
Concentrations in serum samples for 158 endogenous metabolites were determined at IARC by ultra-performance liquid chromatography (LC) (1290 Series HPLC; Agilent, Les Ulis, France) hyphenated to a tandem mass spectrometer (MS/MS) (QTrap 5500; AB Sciex, Les Ulis, France) using the AbsoluteIDQ p180 kit (Biocrates, Innsbruck, Austria). The AbsoluteIDQ p180 Kit (Biocrates, Innsbruck, Austria) is a combined flow injection (FIA) and LC-MS/MS assay. The assay quantifies up to 188 metabolites from five analyte groups: acylcarnitines, amino acids, biogenic amines, hexoses (sum of hexoses), phosphatidylcholines (PCs), and sphingomyelins (SMs). The method combines the derivatization with 5% Phenylisothiocyanate reagent and extraction of analytes using 5mM ammonium acetate in methanol and the selective mass spectrometrical detection using MRM pairs. Isotope-labeled internal standards are partially integrated in the kit plate filter for metabolite quantification. Samples were analyzed using an LC/MS (QTrap 5500; AB Sciex, Les Ulis,France) method (for analysis of amino acids and biogenic amines) followed by FIA-MS (analysis of lipids, acylcarnitines and hexose). The limit of detection for the individual metabolites was set to three times the values of the bufferonly- containing samples. The MetIQ software package (BIOCRATES) allows an automation of the assay workflow, from sample registration to data processing. The AbsoluteIDQ p180 Kit validated by the manufacturer according to the Food and Drug Administration guideline ‘Guidance for industry–Bioanalytical Method Validation, May 2011’. For analytical specifications, refer to the AbsoluteIDQ p180 Kit manuals.
Measurements were made following the procedure recommended by the kit manufacturer. The kit was first tested on EPIC orphan samples, and 158 metabolites out of the proposed 188 were analytically validated. Repeated samples from the same subject were measured within the same analytical batch. Two different quality controls were inserted in each analytical batch. Intra- and inter-batch coefficients of variation ranged from 4% to 15% for the vast majority of analytes. Overall, 4.9% and 1.8% of total values were below the limit of detection (LOD) and the limit of quantification (LOQ), respectively. These values were imputed as LOD and LOQ values, respectively. However, reliability was not assessed when the percentage of values below LOD or LOQ for a given metabolite was greater than 50%, separately in fasting and non-fasting samples, which allowed 140 metabolites to be assessed for reliability.
Variables definition
Samples were defined as fasting samples if time since last food or drink at blood collection was more than six hours. Samples that have been collected less than six hours since last food or drink were considered as non-fasting samples.
Body weight and height were measured at baseline according to standardized procedures previously described [22]. Briefly weight was measured to the nearest 0.1kg and height was measured to the nearest 0.1, 0.5, or 1.0 cm depending on the center, without shoes. The body mass index (BMI) was calculated as body weight in kilograms divided by squared height in meters (kg/m2).
Occupational, recreational and household physical activities were recorded by the EPIC physical activity questionnaire previously described [23]. Total physical activity was estimated as a categorical index (inactive, moderately inactive, moderately active, active) by cross-tabulation of the level of occupational activity (nonworker, sedentary, standing, manual, heavy manual and unknown) with combined recreational and household activities (in quartiles of MET-hours/week).
Other variables such as age (years), smoking status (current, former, never) and time difference between the two blood drawn (years), were also considered in analysis.
Statistical analyses
Concentration levels of biomarkers were log-transformed to approximate normality. Estimates of the intraclass correlation coefficient (ICC) for each metabolite were computed to assess reliability over time [24] as the ratio of between-subject variance component over total variability in mixed models. For each metabolite, the measurement yijk, with i = 1,..,nk indexing study participant, j = 1,2 blood collection occasion and k = 1,2 center, is modelled through a random-effect model as
with α expressing the overall intercept, and β a vector of fixed-effect coefficients to capture the role of confounding variables, notably participants’ age, BMI and the time difference between sample collections. The terms ti(k) and eij(k) are center-specific random-effects study participants and residuals, respectively, whose variance estimates, σ2Bk and σ2Wk, estimate between- and within-subject variability. In this way the center-specific ICC estimates are computed as
. The random-effects models were fitted by using the SAS PROC MIXED procedure which uses a restricted maximum likelihood function for parameter estimation.
The reproducibility of metabolites was assessed according to fasting status, i.e. comparing fasting samples in Turin and non-fasting samples in Utrecht. In order to control for differences of age, BMI and the time difference between sample collections across the two centers, these data were included as fixed effects in the models, whereby smoking status and physical activity had no virtually influence over variance of any of the metabolites. Reliability was considered as excellent with ICCs≥0.75, good with 0.50≤ICCs≤0.74, weak with ICCs<0.50.
The difference in ICC estimates according to fasting status was tested for statistical significance using a bootstrap sampling scheme [25]. A total of 300 repetitions provided sufficiently stable estimates. For each metabolite, P-values were obtained by comparing the ratio of the mean of the ICC difference between fasting and non-fasting samples over the standard deviation of the bootstrap distribution to a standardized normal distribution.
All statistical analyses were performed using SAS 9.2 statistical software (SAS Institute Inc, 2002).
Results
At study entry, compared to women from Utrecht, men from Turin were younger, more frequently current smokers and less physically active, as reported in Table 1.
Reliability over time
Mean levels, within- and between-variance and ICCs estimates of serum metabolites are reported in S1 Table and ICCs are represented in Figs 1 and 2. In fasting samples from Turin, median ICC and ICC range were 0.62 (range: 0–0.89) for acylcarnitines, 0.51 (0.11–0.71) for amino acids, 0.77 (0.54–0.88) for SMs, 0.71 (0.53–0.78) for biogenic amines, and 0.55 for hexoses (Fig 1), and 0.63 (0.39–0.79) for lysoPCs and 0.74 (0.43–0.91) for PCs (Fig 2). Out of 140 metabolites, ICC values were higher than 0.70 for 58 metabolites (41%), and higher than 0.50 for 103 metabolites (74%). None of the hexoses, biogenic amines or SMs compounds showed ICCs<0.5. Among other groups, 48% of amino acids, 27% of acylcarnitines, 18% of lysoPCs and 4% of PCs compounds had ICCs lower than 0.50.
*P-values < 0.05 for difference between fasting and non-fasting ICCs
*P-values < 0.05 for difference between fasting and non-fasting ICCs
In non-fasting samples from Utrecht, median ICC and ICC range were respectively, 0.34 (range: 0.28–0.59) for acylcarnitines, 0.56 (0.23–0.79) for amino acids, 0.54 (0.26–0.66) for SMs, 0.46 (0.06–0.73) for biogenic amines, and 0.58 for hexoses (Fig 1), and 0.62 (0.39–0.77) for lysoPCs and 0.55 (0.20–0.79) for PCs (Fig 2). Out of 140 metabolites, ICC values were higher than 0.7 for 12 metabolites (8%), and higher than 0.5 for 73 metabolites (52%). ICCs<0.5 were seen in 71% of acylcarnitines, 48% of amino acids, 44% of biogenic amines, 36% of SMs, 34% of PCs and 33% of lysoPCs compounds.
In both fasting and non-fasting samples, weak reliability over time was observed in some amino acids, with values ranging from 0.11 to 0.46 for phenylalanine, glutamine, glutamic acid, valine, lysine, histidine, serine, leucine and isoleucine, and a few acylcarnitines compounds, with values lower than 0.46 for C3_Dc_C4_Oh, C5, C3, C16 and C18:1.
Comparison of reliability of fasting vs non-fasting samples
Overall, reliability over time was lower when analyzing non-fasting samples (median ICC: 0.54 with 5th -95th percentile values ranging from 0.26 to 0.75) than fasting samples (0.70; 0.37–0.86). This was particularly apparent for acylcarnitines, PCs and SMs (ICC is lower in non-fasting samples for the majority of them). Reliability of amino acids, biogenic amines, hexoses and lysoPC was overall not different according to fasting status (Figs 1 & 2). ICCs were significantly lower in non-fasting vs. fasting samples in 36% of SMs, 19% of acylcarnitines, 20% of PCs, 17% of biogenic amines, and 0.05% of amino acids (S1 Table).
Discussion
Most of the 140 investigated metabolites showed good reliability over time in serum samples collected on average 2 years apart in participants from the EPIC study. Reliability over time was overall good for non-fasting samples, with a median ICC value equal to 0.54, but metabolites analyzed from fasting samples generally exhibited larger reproducibility (ICCmedian equal to 0.70). Although reproducibility of amino acids, biogenic amines, hexoses and lysoPC did not generally differ according to fasting status, acylcarnitines, PCs and SMs were generally less stable over time for non-fasting compared to fasting samples. It may be noticed that weak reliability over time was observed for some amino acids and acylcarnitines compounds for both fasting and non-fasting samples.
Previous studies have already investigated the reproducibility of endogenous metabolites using the same assay as we used (Biocrates kit) over short time periods in fasting samples [11,12,15]. In 22 healthy German volunteers who donated 3 fasting samples each day, Breier et al. found good reproducibility of most metabolites over a short time period of 3 consecutive days (ICCmedian equal to 0.66), except for long chain and unsaturated acylcarnitines [11]. In 100 EPIC German subjects who gave 2 fasting samples 4 months apart, Floegel et al. reported an overall median ICC of 0.57, with median ICC estimates equal to 0.58 for amino acids, 0.58 for PCs, 0.66 for SMs, 0.76 for hexoses and 0.45 for acylcarnitines [12]. In line with previous findings [11], poor reproducibility in hydroxy- (0.11 ≤ ICCs ≤ 0.45) and monounsaturated acylcarnitines (0.09 ≤ ICCs ≤ 0.63) was mainly attributed to low concentrations [11,12]. In the study by Yu et al. involving 83 repeated fasting samples, most of hydroxy- and unsaturated acylcarnitines were excluded from the analyses because of low concentrations or poor reliability [15]. Among those analytically validated with concentrations higher than LOQ/LOD (16 out of 55), the present study exhibited good reliability of acylcarnitines with a median ICC of 0.62 in fasting samples and good reliability of monounsaturated acylcarnitines (0.46 ≤ ICCs ≤ 0.75). Reproducibility of these 16 acylcarnitine compounds was particularly affected by fasting status, with a median ICC dropping to 0.34 in non-fasting samples. As acylcarnitines occur in the process of fatty acid translocation into the inner mitochondrial membrane for β-oxidation, blood acylcarnitine concentrations reflect the substrate flux through β-oxidation, and then could be particularly affected by the composition and the time since the last meal. The weak reproducibility of phenylalanine, glutamine, glutamic acid, valine, lysine, histidine, serine, leucine and isoleucine amino acids in our study did not confirm previous findings. Amino acids measured in fasting samples generally showed good reproducibility, ICCs ranging from 0.41 to 0.84 in previous studies [11,12]. Amino acids reproducibility was not particularly influenced by fasting status, possibly reflecting the tight genetic regulation of amino acids homeostasis [26].
The main limitation of the present study was the different center origin and gender of fasting and non-fasting samples that led to the comparison of individuals with general characteristics differences which may well have driven part of the difference in reliability. To partially correct for differences in participants’ age, BMI and the time difference between sample collections, a combined mixed model adjusted for these factors was employed. To our knowledge, reproducibility studies on metabolomics profiling have not investigated the effects of gender or country origin on reliability. Yet, the effect of gender on metabolites concentrations seems to be limited as it was estimated to account for 7.3% of variability [14]. The present study included subjects from general population (blood donors and women participating to a breast cancer screening) who were free of disease (diabetes, stroke, heart and overall cancer) at baseline. However, there is the lack of information on health status of subjects at the second time point. If participants had developed chronic conditions during the 2-year follow-up, the within-person variability would increase, leading to underestimation of reliability over time. Only two time points were available in the EPIC cohort for the assessment of metabolites reliability. To study the influence of the number of time points on reliability estimates, Sampson et al. compared ICCs of metabolites calculated from two time points one year apart to those estimated from three time points four years apart [14], and the authors found similar distribution of ICCs. As our study sample was limited in size, no adjustment for multiple testing was adopted for the comparison of fasting and non-fasting ICC estimates. This strategy was chosen in order to avoid non detection of differences between fasting and non-fasting ICC estimates, as the number of metabolites whose reliability was differential with respect to fasting status would have diminished drastically.
The main strength of our study was that we assessed reliability in a wide spectrum of metabolites including different classes of compounds with a validated high-throughput technique that can be applied to future metabolome studies. In particular, our results are likely to be generalizable for the whole EPIC cohort where metabolome measurements have been undertaken. Another strength is that our results showed overall good reliability over a 2-year interval which represents a longer time interval compared to previous investigations involving the same metabolites spectrum [11,12,15]. Finally, to our knowledge, this is the first study that compares reliability in fasting and non-fasting samples for the metabolites measured in this assay (Biocrates kit).
Conclusion
The present study showed good reproducibility for most of the Biocrates metabolites over a 2-year period, including PCs, SMs, biogenic amines, hexoses and most of the amino acids and acylcarnitines in fasting samples from healthy men from the EPIC cohort. More variability over time was observed in the 140 metabolites when measured in non-fasting samples. Our results indicate that a single measurement could be sufficient for hexoses and most of the amino acids, biogenic amines, SMs, and PCs in non-fasting samples, but not for most of the acylcarnitines. Fasting samples are preferable to non-fasting.
Supporting Information
S1 Table. Percentage of samples below the limit of detection (%<LOD), mean concentrations and their 95% confidence intervals (CIs), between- (B) and within-subject (W) variance components, intraclass correlation coefficients (ICC) of serum concentrations measured in 66 healthy subjects from two EPIC centers.
P-values for difference in fasting and non-fasting samples are also reported.
https://doi.org/10.1371/journal.pone.0135437.s001
(XLSX)
Author Contributions
Conceived and designed the experiments: SR AS PF TJK NCOM PV CS. Performed the experiments: DA. Analyzed the data: PF IL MC. Wrote the paper: MC IL PF. Designed the statistical plan: PF. Reviewed and provided substantial revision to the manuscript: SR AS TJK PV.
References
- 1. Dettmer K, Aronov PA, Hammock BD (2007) Mass spectrometry-based metabolomics. Mass Spectrom Rev 26: 51–78. pmid:16921475
- 2. Dunn WB, Broadhurst DI, Atherton HJ, Goodacre R, Griffin JL (2011) Systems level studies of mammalian metabolomes: the roles of mass spectrometry and nuclear magnetic resonance spectroscopy. Chem Soc Rev 40: 387–426. pmid:20717559
- 3. Mayers JR, Wu C, Clish CB, Kraft P, Torrence ME, et al. (2014) Elevation of circulating branched-chain amino acids is an early event in human pancreatic adenocarcinoma development. Nat Med 20: 1193–1198. pmid:25261994
- 4. Wang TJ, Larson MG, Vasan RS, Cheng S, Rhee EP, et al. (2011) Metabolite profiles and the risk of developing diabetes. Nat Med 17: 448–453. pmid:21423183
- 5. Wurtz P, Makinen VP, Soininen P, Kangas AJ, Tukiainen T, et al. (2012) Metabolic signatures of insulin resistance in 7,098 young adults. Diabetes 61: 1372–1380. pmid:22511205
- 6. Scalbert A, Brennan L, Fiehn O, Hankemeier T, Kristal BS, et al. (2009) Mass-spectrometry-based metabolomics: limitations and recommendations for future progress with particular focus on nutrition research. Metabolomics 5: 435–458. pmid:20046865
- 7. Jenab M, Slimani N, Bictash M, Ferrari P, Bingham SA (2009) Biomarkers in nutritional epidemiology: applications, needs and new horizons. Hum Genet 125: 507–525. pmid:19357868
- 8.
Rappaport SM, Kupper LL (2008). Quantitative exposure assessment. S. Rappaport, El Cerrito, Calif.
- 9.
Fleiss JL (1999) Reliability of Measurement. The Design and Analysis of Clinical Experiments: John Wiley & Sons, Inc. pp. 1–32.
- 10. Rosner B, Willett WC, Spiegelman D (1989) Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error. Stat Med 8: 1051–1069. pmid:2799131
- 11. Breier M, Wahl S, Prehn C, Fugmann M, Ferrari U, et al. (2014) Targeted metabolomics identifies reliable and stable metabolites in human serum and plasma samples. PLoS One 9: e89728. pmid:24586991
- 12. Floegel A, Drogan D, Wang-Sattler R, Prehn C, Illig T, et al. (2011) Reliability of serum metabolite concentrations over a 4-month period using a targeted metabolomic approach. PLoS One 6: e21103. pmid:21698256
- 13. Nicholson G, Rantalainen M, Maher AD, Li JV, Malmodin D, et al. (2011) Human metabolic profiles are stably controlled by genetic and environmental variation. Mol Syst Biol 7: 525. pmid:21878913
- 14. Sampson JN, Boca SM, Shu XO, Stolzenberg-Solomon RZ, Matthews CE, et al. (2013) Metabolomics in epidemiology: sources of variability in metabolite measurements and implications. Cancer Epidemiol Biomarkers Prev 22: 631–640. pmid:23396963
- 15. Yu Z, Kastenmuller G, He Y, Belcredi P, Moller G, et al. (2011) Differences between human plasma and serum metabolite profiles. PLoS One 6: e21230. pmid:21760889
- 16. Cross AJ, Moore SC, Boca S, Huang WY, Xiong X, et al. (2014) A prospective study of serum metabolites and colorectal cancer risk. Cancer 120: 3049–3057. pmid:24894841
- 17. Moore SC, Matthews CE, Sampson JN, Stolzenberg-Solomon RZ, Zheng W, et al. (2014) Human metabolic correlates of body mass index. Metabolomics 10: 259–269. pmid:25254000
- 18. Medina S, Domínguez-Perles R, Ferreres F, Tomás-Barberán FA, Gil-Izquierdo Á (2013) The effects of the intake of plant foods on the human metabolome. TrAC Trends in Analytical Chemistry 52: 88–99.
- 19. Riboli E, Hunt KJ, Slimani N, Ferrari P, Norat T, et al. (2002) European Prospective Investigation into Cancer and Nutrition (EPIC): study populations and data collection. Public Health Nutr 5: 1113–1124. pmid:12639222
- 20. Leenders M, Ros MM, Sluijs I, Boshuizen HC, van Gils CH, et al. (2013) Reliability of selected antioxidants and compounds involved in one-carbon metabolism in two Dutch cohorts. Nutr Cancer 65: 17–24. pmid:23368909
- 21. Palli D, Berrino F, Vineis P, Tumino R, Panico S, et al. (2003) A molecular epidemiology project on diet and cancer: the EPIC-Italy Prospective Study. Design and baseline characteristics of participants. Tumori 89: 586–593. pmid:14870823
- 22. Haftenberger M, Lahmann PH, Panico S, Gonzalez CA, Seidell JC, et al. (2002) Overweight, obesity and fat distribution in 50- to 64-year-old participants in the European Prospective Investigation into Cancer and Nutrition (EPIC). Public Health Nutr 5: 1147–1162. pmid:12639224
- 23. Cust AE, Smith BJ, Chau J, van der Ploeg HP, Friedenreich CM, et al. (2008) Validity and repeatability of the EPIC physical activity questionnaire: a validation study using accelerometers as an objective measure. International Journal of Behavioral Nutrition and Physical Activity 5: 33. pmid:18513450
- 24.
Dunn G (1989) Design and analysis of reliability studies: The statistical evaluation of measurement errors: Edward Arnold Publishers.
- 25.
Efron B, Tibshirani RJ (1994) An introduction to the bootstrap: CRC press.
- 26. McBride KL, Belmont JW, O'Brien WE, Amin TJ, Carter S, et al. (2007) Heritability of plasma amino acid levels in different nutritional states. Mol Genet Metab 90: 217–220. pmid:17005426