Development and Validation of a Dementia Risk Prediction Model in the General Population: An Analysis of Three Longitudinal Studies

Reliable identification of individuals at increased risk of dementia is essential for individualized risk management in both primary and specialized clinical care, but also for optimal design of preventive trials (1). This necessity was aptly demonstrated by recent findings from large randomized controlled trials that showed potential efficacy of multidomain interventions to prevent cognitive decline in high-risk individuals (2–5). The Finnish Geriatric Intervention Study to Prevent Cognitive Impairment and Disability (FINGER) trial (2) showed that a multidomain lifestyle intervention resulted in a significant protective effect on cognition. The success of this trial has been attributed in part to the tailored approach of targeting only an at-risk segment of the general population for these preventive interventions (2). This strategy was further corroborated by secondary analyses from the Prevention of Dementia by Intensive Vascular Care (PreDIVA) trial (3) demonstrating that intensive vascular risk management had the strongest effect among participants with untreated hypertension (3). It is now increasingly recognized that such preventive strategies may be most effective in an at-risk population (3, 4, 6–8).

Several models have been developed to predict dementia in the general population (9), but external validation recently showed that these have limited incremental predictive value above and beyond age (10). These models were mostly based on lifestyle factors, social factors, and comorbidities. So far, models are lacking that include information on markers that reflect the underlying disease process, especially in its early stages. Such markers include subjective memory decline, APOE genotype, and neuroimaging (11–14). On the other hand, such markers are usually not available in primary care settings. It is therefore conceivable that different models are required, depending on the setting: simple non-laboratory models for primary care settings and extended biomarker-based models for specialized clinical care settings. Note, however, that for the purposes of risk stratification in healthy individuals in primary care settings, models should preferably be based on risk factors that can be obtained without invasive diagnostics such as CSF sampling or imaging requiring substantial amounts of ionizing radiation such as positron emission tomography (PET).

Another important consideration is that dementia prediction models should take into account the competing risk of death from other causes, given the generally late-life onset of dementia among community-dwelling individuals. Failure to account for such competing risks inflates apparent dementia risk predictions, limiting the practical utility of currently available models (15).

Our aim in this study was to develop a dementia prediction model for use in primary care settings. We also examined whether an extended model that included cognitive, genetic, and imaging markers could improve predictive performance. Both models were developed while accounting for competing risks.

Methods

Study Population

This study was embedded in the Rotterdam Study, a prospective population-based cohort study (16). Since 1990, inhabitants age 55 and older residing in Ommoord, a district of Rotterdam, the Netherlands, have been invited to participate in the study. Of the 20,744 invited inhabitants, 14,926 (72%) agreed to participate. Follow-up examinations take place every 3 to 4 years. In addition, a random sample of Rotterdam Study participants were invited for brain MRI scanning in 1995 and 1996 (N=563). From 2005 onward, brain MRI became part of the core study protocol of the Rotterdam Study (17). For the present study, we selected participants age 60 or older for whom baseline data were available on clinical, cognitive, genetic, and MRI parameters (see the flowchart of study participants in Figure S1 in the online supplement). We excluded participants who had dementia or incomplete screening for dementia at baseline (N=40), those who did not provide informed consent to access medical records (N=11), and those for whom no follow-up was available for logistical reasons (N=35). We also excluded participants for whom valid imaging data were not available because of artifacts or other reasons (e.g., contraindications to MRI or signs of claustrophobia during acquisition) (N=124) as well as those for whom data were missing on APOE status (N=134). After exclusions, a total of 2,710 participants were included for analysis in this study, all of whom provided written informed consent to participate in the study and to have their information obtained from treating physicians.

Candidate Predictors

Detailed methods on predictor data collection and predictor definitions are described in the online supplement. We prespecified candidate predictors on the basis of the literature, expert knowledge, and availability in clinical practice. For a primary care model, we considered the following candidate predictors: age, sex, education level, systolic blood pressure, smoking, history of diabetes, history of stroke, presence of depressive symptoms, parental history of dementia, presence of subjective memory decline, and need for assistance with finances or medication. For the extended model, we considered the addition of cognitive tests (word fluency test, letter digit substitution test, Stroop interference, and delayed word learning test), APOE-ε4 genotype, and brain MRI parameters (white matter hyperintensity volume, total brain volume, hippocampal volume, and presence of infarcts [lacunar/cortical]). White matter hyperintensity, total brain, and hippocampal volume were all entered into the models as a percentage of intracranial volume to correct for differences in head size.

Assessment of Dementia

Participants were screened in-person for dementia at baseline and at subsequent center visits with the Mini-Mental State Examination (MMSE) and the Geriatric Mental State Schedule organic level (18). Those with an MMSE score <26 or a Geriatric Mental State Schedule score >0 underwent further investigation and informant interview, including the Cambridge Examination for Mental Disorders of the Elderly. The information from in-person screening was supplemented by data from the electronic linkage of the study database with medical records from all general practitioners and the regional institute for outpatient mental health care. In the Dutch health care system, the entire population is entitled to primary care that is covered by their (obligatory) health insurance. The entire cohort is thus continuously monitored for detection of interval cases of dementia or cognitive disturbances between center visits. Study physicians evaluate all records biannually and combine information from medical records with in-person screening to draw up individual case reports. In these reports, the physicians covered all gathered relevant information to establish the presence, probability, and subtype of dementia. A consensus panel led by a consultant neurologist established the final diagnosis according to standard criteria for dementia (DSM-III-R) and Alzheimer’s disease (National Institute of Neurological and Communicative Disorders and Stroke–Alzheimer’s Disease and Related Disorders Association). All participants were followed for incident dementia until January 1, 2015. Follow-up was virtually complete (97.2% of potential person-years).

External Validation

For external validation of the models, we used the Epidemiological Prevention Study of Zoetermeer (EPOZ) from the Netherlands and the Alzheimer’s Disease Neuroimaging Initiative cohort 1 (ADNI-1) from the United States. EPOZ started in 1975, with the aim of assessing the prevalence of several chronic diseases and their determinants in the city of Zoetermeer (19). Response rates were similar to those of the Rotterdam Study (72%). In 1995 and 1996, a random subsample of the participants who were between ages 60 and 90 underwent cognitive testing and brain MRI (N=514); data from this group are considered the baseline for the present study. Participants were screened at study entry and at follow-up visits for dementia, using a strict protocol (20). All participants were followed for incident dementia until the end of study, on January 1, 2007 (follow-up included 90.8% of potential person-years). For validation within ADNI, we selected 228 cognitively unimpaired individuals age 60 or older. Data used in the preparation of this study were obtained from the ADNI database (adni.loni.usc.edu). The primary goal of ADNI has been to test whether serial MRI, PET, other biological markers, and clinical and neuropsychological assessments can be combined to measure the progression of mild cognitive impairment and early Alzheimer’s disease. Further details on ADNI can be found elsewhere (21).

Statistical Analysis

To reduce extreme effects of the predictors, we truncated the distribution of continuous variables at the 1st and 99th percentiles. Distributions for white matter hyperintensity volume and Stroop interference score were skewed. We obtained normal distributions of these parameters using a natural logarithmic transformation. We modeled potential nonlinear effects of age by using restricted cubic spline transformations and by adding an age-squared term, to capture most accurately the effects of age as the most important risk factor for dementia.

For the basic model, we used competing risk regression as proposed by Fine and Gray (22), with all candidate predictors included and fitted into the model to calculate 10-year risk of dementia. Table S2 in the online supplement provides further details on the development steps of the model and testing of the assumptions. We subsequently used the least absolute shrinkage and selection operator (LASSO) technique, adapted to a competing risk setting to simultaneously penalize the model’s regression coefficients and select important predictors for the final model (23, 24). The LASSO method is particularly useful for preventing model overfitting and model misspecification (25). An overfitted model tends to underestimate the probability of an event in low-risk groups and overestimate an event in high-risk groups.

For the development of the extended model, we used the predictors selected by the LASSO technique in the basic model as a starting point and extended it with the addition of objective cognitive tests, APOE-ε4 carrier status, and brain MRI parameters. As a reference, we used a model based on age alone for all analyses. In a stepwise exploratory analysis, we investigated the additive predictive value for each domain separately (cognition, imaging, and genetic information) of the final extended model, compared with the basic model. All of the C-statistics from the development sample presented here are corrected for optimism.

Internal Validation

We evaluated the robustness of the model using bootstrap samples for each model and found consistent results in selection steps and coefficient shrinkage using the LASSO technique, based on 200 bootstrap samples (see Tables S4 and S6 in the online supplement) (26). We quantified the discriminative ability of these models using the C-statistic for survival data with competing outcomes (27, 28). The C-statistic is an adapted area-under-the-curve metric for use in survival analyses. It indicates the overall proportion of all pairs of participants that can be ordered such that the participant who developed dementia during follow-up indeed had a higher predicted risk. We used the cumulative incidence function to calculate the absolute 10-year risk of dementia (29). We used the DeLong test adapted for survival analyses to infer whether C-statistics of the basic and extended models were statistically different from those of a model based on age alone (30).

Stratified Analysis

We assessed the predictive accuracy for the most common subtype of dementia, namely, Alzheimer’s disease, and assessed model performance for men and women separately. Next, we excluded the first 4 years of follow-up to assess whether the predictive value extended beyond the first years of follow-up, since some of the predictors may reflect prodromal or undiagnosed dementia. To further investigate model robustness across varying time horizons, we evaluated the predictive ability of the model using 3-, 5-, and 15-year time horizons. Finally, we stratified on age (≤80 years and >80 years) at baseline, given the median age at diagnosis (31) and the steep increase in incidence of dementia beyond this age in order to investigate the performance of the model at different ages.

Missing data on predictors were imputed using multiple imputation, based on all predictors, outcome status, and follow-up time. All analyses were performed using R, CRAN version 3.3.2 (with the rms, cmprsk, mycrr [27], and crrp [24] packages).

Results

Development Cohort Study Population

The baseline characteristics of the 2,710 participants of the development cohort are summarized in Table 1. The mean age was 71.2 years; 52.8% of the participants were women, and 33.3% had subjective memory decline. During a median follow-up of 7.0 years for those who were censored alive (interquartile range, 5.1–9.1), with a total follow-up of 20,324 person-years, 181 participants developed dementia, of whom 146 developed Alzheimer’s disease; 578 participants died from other causes. This corresponds to a crude incidence rate for dementia of 9.2 per 1,000 person-years. During the 10-year predicted time horizon, 131 participants developed dementia and 444 participants died free of dementia.

Model Development and Internal Validation

There was evidence for a nonlinear relationship between age and risk of dementia. We therefore added age-squared into the model to capture this nonlinearity. The basic model considered age, age-squared, sex, education level, systolic blood pressure, current smoking, history of diabetes, history of symptomatic stroke, depressive symptoms, parental history of dementia, presence of subjective memory decline, and need for assistance with finances or medication. In Table 2, the subdistribution hazard ratios are presented.

The discriminative accuracy measured with the C-statistic of the full basic model was 0.79 (95% CI=0.76, 0.83). After shrinkage and predictor selection using the LASSO technique, all statistically significant predictors remained in the model: age, history of symptomatic stroke, presence of subjective memory decline, and need for assistance with finances or medication. The C-statistic remained similar, at 0.79 (95% CI=0.76, 0.82). Based on 200 bootstrap samples, model optimism was small (that is, the predictive performance of the models was not tied to a specific sample; the optimism-corrected C-statistic for the basic model was 0.78 [95% CI=0.75, 0.81]; see Tables S4 and S6 in the online supplement). From here, all presented C-statistics derived from the development study are corrected for optimism to represent optimism-corrected C-statistics.

Adding cognitive, APOE-ε4 carrier status, and imaging information to the basic model resulted in higher discriminative ability (C-statistic=0.86, 95% CI=0.83, 0.88). In Table S5 in the online supplement, C-statistics are presented when cognitive, APOE-ε4 carrier status, or imaging information are added to the basic model separately. After shrinkage and selection, the letter digit substitution test, the delayed word learning test, APOE-ε4 carrier status, and all imaging markers except brain infarcts were selected (C-statistic=0.86, 95% CI=0.83, 0.88). Ten-year risks based on the basic model are easily calculated using a simple risk chart (Figure 1). An Excel spreadsheet to calculate risks with the extended model is available as an online supplement.

Stratified Analyses

The basic and extended models showed roughly similar results for Alzheimer’s disease, and for men and women separately (Table 3). Discriminative ability remained similar when the first 4 years of follow-up were excluded (C-statistic=0.79, 95% CI=0.74, 0.84). Using 3- and 5-year predicted time horizons, the basic and extended models had higher discriminative properties (the C-statistics for the basic and extended models, respectively, were 0.82 [95% CI=0.78, 0.86] and 0.91 [95% CI=0.87, 0.95] for a 3-year horizon and 0.79 [95% CI=0.74, 0.83] and 0.88 [95% CI=0.85, 0.91] for a 5-year horizon), compared with the 10-year predicted time horizon (Table 4). In contrast, when using a 15-year predicted time horizon, the basic and extended models had lower discriminative properties (the C-statistics were 0.67 [95% CI=0.62, 0.74] and 0.71 [95% CI=0.65, 0.75], respectively). In individuals older than age 80 (N=456), the basic model showed considerably lower discriminative ability (C-statistic=0.57, 95% CI=0.49, 0.63). In contrast, the extended model retained substantial discriminative ability in this stratum (C-statistic=0.71, 95% CI=0.64, 0.76) and was considerably higher compared to a prediction based on age alone (C-statistic=0.53, 95% CI=0.45, 0.63).

External Validation

The baseline characteristics of the Rotterdam Study and EPOZ participants were largely similar, whereas the ADNI-1 participants, on average, were older, attainted a higher education level, reported less memory decline, and were more likely to have a history of parental dementia (Table 1). During a median of 9.5 years (interquartile range, 7.6–11.4) of follow-up in EPOZ, 36 participants developed dementia and 120 participants died free of dementia. During a median follow-up time of 6.3 years (interquartile range, 2.0–8.0) in ADNI-1, 26 participants developed dementia. Within EPOZ, both the basic and the extended model showed discriminative performance in line with their performance in the development cohort (the C-statistics were 0.75 [95% CI=0.67, 0.82] for the basic model and 0.81 [95% CI=0.74, 0.88] for the extended model). The models were well calibrated (see Figure S2 in the online supplement). As reference, a model based on age alone yielded a C-statistic of 0.73 (95% CI=0.65, 0.82) and resulted in significantly worse performance compared with the basic and extended models (p values <0.001 for both). Given that ADNI is not a population-based study and recruits participants through specialized clinical study sites, we only tested the performance of the full model. This yielded a lower C-statistic of 0.72 (95% CI=0.63, 0.83), reflecting a more homogeneous and older population, yet it also performed significantly better than a model based on age alone (C-statistic=0.54, 95% CI=0.42, 0.64, p=0.01).

Discussion

In this study, we developed and validated a simple prediction model for dementia in an aging population in primary care. In addition, we demonstrated that this performance can be further extended in a model including cognitive testing, APOE genotyping, and brain MRI parameters. These models can be used to calculate the 10-year risk of dementia to inform individuals and optimize risk stratification for clinical trials.

The discriminative ability of our basic model was comparable to previously published models incorporating data for use in primary care settings (9). Most previous studies only reported on discriminative ability, ranging from 0.65 to 0.80 as measured with the C-statistic. For instance, the Brief Dementia Screening Indicator, using data available in primary care, yielded C-statistics between 0.68 and 0.78 across four cohorts. Notably, four other prediction models included in a recent external validation study did not provide additional predictive value in dementia risk prediction compared with a model with age as the only predictor (10). In the present study, the basic model we developed did show greater discriminative ability and improved calibration above and beyond age alone. Compared with the Brief Dementia Screening Indicator model, our basic model additionally included the presence of subjective memory decline. The strength of this predictor in relation to the occurrence of dementia (adjusted hazard ratio=1.65) and the prevalence in the general population (33%) resulted in better predictive performance.

The models in this study include a history of stroke instead of the various individual cardiovascular risk factors included in several previous models (9). We did consider traditional cardiovascular risk factors, but they did not pass the mark for inclusion in the final models. Several explanations may underlie these observations. First, almost a quarter of all dementia cases can be attributed to vascular risk factors, illustrating their etiological importance in the development of dementia (18, 32, 33). However, similar to coronary heart disease prediction in the elderly (34, 35), the role of cardiovascular risk factors in dementia prediction may strongly diminish with age. Second, cardiovascular risk factors are also strongly associated with various other diseases in old age, reducing their specific discriminative ability in predicting the occurrence of dementia. For instance, smoking could lead to potentially fatal competing events, such as cardiovascular events or cancer at younger ages, and thereby preclude the occurrence of dementia. As a consequence, smoking has limited specificity in predicting cardiovascular disease, cancer, or dementia at older ages. Dementia risk prediction models should take into account competing risks to avoid uninterpretable C-statistics and inflated absolute risks (15). We dealt with this issue in this study by deriving our dementia prediction models within a competing risk framework.

In line with results from a previous model based on predictors derived in a primary care setting (36), our basic model had poor discriminative ability in individuals older than age 80. This finding is generally of limited concern when using a prediction model to identify high-risk individuals for clinical trials, since trials generally aim to recruit younger individuals. Yet, these findings provide insight into the complexity of dementia prediction using only clinical parameters in the oldest-old. In contrast, our extended model showed substantially higher discriminative ability for individuals older than age 80, highlighting the significance of objective markers of cognition and brain structure in the oldest-old, including cognitive testing, genetics, and brain imaging.

In this study, we developed and validated two complementary risk models: a basic model that could be used by family doctors and general practitioners, and an extended model that could be used in a specialized clinical care setting and that incorporates cognitive testing, brain MRI parameters, and genetic data. The strength of the extended model is that it uses information that reflects the underlying disease process. At the same time, it can be argued that the presence of these markers indicates that the disease is already present, raising the question whether the model offers prediction or in fact early diagnosis. Nevertheless, our sensitivity analyses excluding the first 4 years of follow-up showed similar predictive accuracies, suggesting that the effect of early diagnosis as opposed to prediction was marginal. Moreover, the ability to identify persons 10 years before clinical diagnosis can inform trials aimed at intervening in the earliest phase.

Indeed, it is now increasingly recognized that preventive or treatment strategies may be more effective when targeting individuals who are at increased risk of dementia (1, 6–9, 37). In order to direct such interventions to those who would most likely benefit from them, a reliable way to identify individuals at high risk for dementia is needed. The prediction models presented here address this gap, and they can be used to stratify individuals in clinical trials. Absolute 10-year dementia risk thresholds for determining low- and high-risk groups need to be established and may depend on the research question at hand, as well as the availability, costs, and risks of the intervention. These models can be combined in a two-step design, providing opportunities to identify at-risk individuals from the general population in primary care with a simple yet predictive model. Subsequently, these individuals can be referred to a more specialized care setting, where a more refined risk assessment can be done using the extended model. It would be interesting to investigate whether the performance of the basic model could be further improved with the addition of a simple blood test (38), or a brief cognitive test, such as the visual association test (39). The extended model could be further improved by adding novel imaging modalities, such as cerebral microbleeds or data on diffusor tensor imaging of the brain, by including rare genetic variants and functional genomics, or by extending the model with more in-depth neuropsychological tests (40–42). The predictive value of other predictors that were available either in the Rotterdam Study or in the validation studies could have been interesting to explore. However, in this study, we specifically aimed to develop a dementia prediction model and to validate exactly that model in these validation studies. Exploring the predictive yield of additional predictors would technically lead to the development or extension of another prediction model, which subsequently would have to be externally validated again.

The following limitations of this study must be considered. First, we used a regularization method (LASSO) that automatically selects and subsequently shrinks effect sizes of important predictors. This penalization strategy may have led to some underestimation of predictor effects in the development sample, but it increases the likelihood of replication in validation studies. Second, this study focused on older adults of predominantly Caucasian descent (>97%). Therefore, these models may not generalize to younger individuals or other ethnicities, and further validation work in these groups is needed. Third, we developed the models in a population-based setting, which matches the primary care setting, but this will likely affect model performance when validated or used in selected populations seen in specialized clinical care. This was in part reflected by a slightly lower discriminative accuracy in ADNI, yet in addition to differences in case mix (including the homogeneous character of this highly selected sample) and a relatively high attrition rate, discrimination remained substantially better than in a model based on age alone. Fourth, we used data on brain imaging with quantitative parameters, which may influence model performance compared with qualitative analyses, such as atrophy and white matter hyperintensity scales. Finally, dementia prediction without an effective therapy at hand raises ethical concerns. While such models are unlikely to be rolled out into clinical practice before further validation and assessment are undertaken, they have been shown to be useful for selecting individuals into clinical trials (2).

Strengths of the study include the large sample size and the availability of detailed information on a wide selection of potential dementia predictors. Moreover, the basic model is based on questionnaire information and is therefore simple to use, and it requires no further testing or laboratory measurements. Finally, the models were well validated, both internally and externally.

Conclusions

In this study, we developed and validated a dementia prediction model providing accurate dementia risk stratification and estimation in a general aging population. Addition of cognitive, imaging, and genetic features improved the predictive ability. These models can be used to identify individuals at high risk for dementia in the general population and may inform future clinical trial design.