Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A Systematic Review of Predictions of Survival in Palliative Care: How Accurate Are Clinicians and Who Are the Experts?

Abstract

Background

Prognostic accuracy in palliative care is valued by patients, carers, and healthcare professionals. Previous reviews suggest clinicians are inaccurate at survival estimates, but have only reported the accuracy of estimates on patients with a cancer diagnosis.

Objectives

To examine the accuracy of clinicians’ estimates of survival and to determine if any clinical profession is better at doing so than another.

Data Sources

MEDLINE, Embase, CINAHL, and the Cochrane Database of Systematic Reviews and Trials. All databases were searched from the start of the database up to June 2015. Reference lists of eligible articles were also checked.

Eligibility Criteria

Inclusion criteria: patients over 18, palliative population and setting, quantifiable estimate based on real patients, full publication written in English. Exclusion criteria: if the estimate was following an intervention, such as surgery, or the patient was artificially ventilated or in intensive care.

Study Appraisal and Synthesis Methods

A quality assessment was completed with the QUIPS tool. Data on the reported accuracy of estimates and information about the clinicians were extracted. Studies were grouped by type of estimate: categorical (the clinician had a predetermined list of outcomes to choose from), continuous (open-ended estimate), or probabilistic (likelihood of surviving a particular time frame).

Results

4,642 records were identified; 42 studies fully met the review criteria. Wide variation was shown with categorical estimates (range 23% to 78%) and continuous estimates ranged between an underestimate of 86 days to an overestimate of 93 days. The four papers which used probabilistic estimates tended to show greater accuracy (c-statistics of 0.74–0.78). Information available about the clinicians providing the estimates was limited. Overall, there was no clear “expert” subgroup of clinicians identified.

Limitations

High heterogeneity limited the analyses possible and prevented an overall accuracy being reported. Data were extracted using a standardised tool, by one reviewer, which could have introduced bias. Devising search terms for prognostic studies is challenging. Every attempt was made to devise search terms that were sufficiently sensitive to detect all prognostic studies; however, it remains possible that some studies were not identified.

Conclusion

Studies of prognostic accuracy in palliative care are heterogeneous, but the evidence suggests that clinicians’ predictions are frequently inaccurate. No sub-group of clinicians was consistently shown to be more accurate than any other.

Implications of Key Findings

Further research is needed to understand how clinical predictions are formulated and how their accuracy can be improved.

Introduction

Studies show that patients, carers, and clinicians all value accurate prognostic information [16]. Prognostic accuracy is important at all stages of the illness trajectory [7]. When a prognosis is discussed openly, it can give family members, patients, and clinicians the opportunity to engage fully with each other, make informed decisions and receive specialist physical and emotional support in a timely manner [7, 8], particularly when the prognosis is short.

In the United Kingdom, a recent review of a care pathway for a dying patient called the Liverpool Care Pathway (LCP) [9], highlighted that clinicians are not very accurate at recognising which patients are imminently dying. This is in contrast to previous research which has suggested an “horizon effect” in prognostication [10]. The so-called “horizon effect” suggests that clinicians should be more accurate at recognising a shorter rather than a longer prognosis.

There have been three reviews published that have reported on the accuracy of clinician estimates which suggest that clinicians’ predictions about length of survival are inaccurate and unreliable [1012]. These reviews have all been limited to patients with advanced cancer. Evidence for patients with a non-cancer diagnosis suggests that clinicians’ determinations of prognosis in these patients may be more inaccurate than those in cancer patients [13].

The most common method of predicting survival in clinical practice remains simple clinical intuition. In order to improve general clinicians’ prognostic skills it is important to learn from clinicians who have a particular expertise in this area. Which leads to the questions, are some clinicians better at prognosticating than others? Are there individual factors, such as professional training or years of experience that make a clinician a more expert prognosticator?

This review extends current literature by including all diagnoses and including all healthcare professionals. Using this approach, our final conclusion should be applicable to all disciplines who are asked to provide a prognosis.

Aims

The systematic review questions were:

  • How accurate are clinicians’ predictions of survival in palliative care patients?
  • Are any subsets of clinicians more “expert” at prognostication than others?

Methods

The protocol for this systematic review is available as supplementary material (S1 Appendix)

Search Strategy

The search strategy was developed in line with the recommendations of the Cochrane Prognosis Methods Group [14]. The search strategies from previous literature [11, 15] were also referred to for guidance. Combined terms used were for: “Palliative care patients”; “Clinicians’ predictions”; and “Prognosis” (S2 Appendix). Sensitivity of the search strategy was tested by running the search and checking that key papers known to the authors were identified.

The databases searched were MEDLINE, Embase, CINAHL, and the Cochrane Database of Systematic Reviews and Trials. Searches were conducted from inception up to June 2015. A search of the reference lists of the final studies was also conducted.

Authors identified in the review were contacted and asked if they were aware of any unpublished literature in the area. A grey literature website [16] was searched for unpublished work.

Inclusion/exclusion criteria

Inclusion.

Studies were included in this review if all the following criteria were satisfied:

  • Patients were over 18
  • Patients were defined within the study as being “not curative”, “palliative”, or having a “terminal illness”
  • The clinician making the prognostic estimate worked in a palliative care setting (i.e. a hospital or community palliative care team, or a hospice). A clinician, in this review, was defined as healthcare professional, such as a doctor (of any profession), a nurse, or any clinician who provides therapeutic support to a patient.
  • Any study design in which a prognosis from a clinician was quantified either in terms of duration or probability of survival
  • Written in English

Exclusion.

Studies were excluded if any of the following criteria were satisfied:

  • Animal study
  • Age of the patients was less than 18 years
  • The clinical setting was Intensive Care Unit (or similar) or patients were receiving artificial ventilation
  • The study concerned assessment of prognosis following a specific intervention e.g. survival following surgery or chemotherapy
  • Only published in abstract form
  • The prognostic estimates were based on hypothetical cases rather than real patients.
  • The prognostic questions were not quantifiable (e.g. I would not be surprised if this patient died within one year).

Quality Assessment

Identifying prognostic studies and evaluating their risk of bias is challenging [15, 17]. We used the QUIPS tool to assess bias [18]. The domain of “Study Participation” was scored twice, in order to reflect the involvement of both clinician and patient populations within the same study. The tool was completed by one researcher (NW). In the event of any doubt about the score, an independent second reviewer (PS) discussed the study with the researcher.

It was decided that no study would be excluded based on the quality assessment score, in order to provide a full account of clinician survival estimates. For several of the studies identified, the accuracy of the clinical estimate of prognosis was not the primary outcome of the research, but was part of a secondary analysis. The QUIPS score of each paper has been reported for transparency but has not been used as a basis for exclusion.

Data Extraction

Using a standardised table, one reviewer (NW) extracted information from each study regarding the setting, characteristics of clinicians, type of prognostic estimate (see below), and patient population. In the event of uncertainty, a second reviewer (PS) was consulted. In order to facilitate synthesis of data, studies were grouped according to the type of prognostic estimate obtained; categorical, continuous, or probabilistic (see below for definitions).

Categorical prognostic estimates.

Categorical prognostic estimates occurred when clinicians were asked to pick from a pre-determined list of survival durations, e.g. 0–14 days, 15–56 days and >56 days, or the analysis had been reported using such categories. The raw data from each study were extracted; where percentage accuracy was given, the absolute number was calculated. The number of accurate estimates relative to the total number of estimates provided in the study was calculated. Accuracy, in this context, equates to the frequency with which the clinician selected the correct survival category.

Continuous prognostic estimates.

Continuous prognostic estimates occurred when clinicians were asked an open question about how long a patient was expected to survive (e.g. how many days do you expect this patient to live?). The data from these studies were often reported as the median predicted and median actual survival. The outcome was usually reported in days, however in several papers, weeks were recorded. In order to keep the outcome the same across the studies, all estimates were converted to days. Accuracy, in this context, is defined as the difference between median predicted and median actual survival.

Probabilistic prognostic estimates.

Probabilistic prognostic estimates occurred when clinicians were asked to determine the percentage likelihood of an outcome at a specified time-point (e.g. what is the probability that this patient will be alive in three months’ time?).

Relative prognostic accuracy of different types of clinicians.

Information about the clinicians being evaluated (e.g. professional background, speciality training, and years of experience) and the types of prognostic estimate they were asked to undertake (categorical, continuous or probabilistic) were extracted where possible. Further categorisation by years of certification or speciality was not possible due to a lack of available information.

Missing data.

When data were not presented fully in published reports, the study authors were contacted for more information [1937]; three authors returned additional data [3537]. For numerous studies providing continuous estimates, predicted and actual survival data were missing, and were no longer available from the study authors [22, 28, 29, 38, 39]. In these cases, we used summary results presented in a previous systematic review [11] which had managed to obtain the raw data before it had been destroyed. For those studies where the relevant data could not be obtained, the results were presented narratively.

Data analysis

For studies with categorical prognostic estimates, a forest plot was created showing the accuracy of estimates as a percentage of the total number of estimates for each study. For studies with continuous prognostic estimates, a Professional Error Score (PES) was calculated for each study. The PES is the difference between the median predicted survival (PS) and the median actual survival (AS) as a percentage of the actual survival; (PS–AS)/AS)*100 [27], where 0 represents perfect accuracy. For studies with a probabilistic estimate, the data were described narratively.

Several papers presented accuracy in terms of the area under the Receiver Operating Characteristic (ROC), known as the c-statistic or the ‘ROC value’. These analyses are frequently used when assessing the accuracy of a diagnostic test. True positive rates (sensitivity) are plotted against false positive rates (1—specificity) to investigate whether clinicians can discriminate accurately between those who will and won’t die at particular time points. The closer the ROC value or c-statistic is to 1, the more accurate are the clinicians. As a general rule, a value of 0.5 suggests no ability to discriminate, a value of ≥0.7 and <0.8 an acceptable level of discrimination, ≥0.8 and <0.9 an excellent level of discrimination and ≥0.9 is outstanding.

Due to the degree of clinical heterogeneity between studies, it was deemed inadvisable to conduct a meta-analysis to calculate a pooled “overall” estimate, for any of the types of estimate considered (categorical, continuous or probabilistic).

STATA v13 was used for the data analyses.

Results

A summary of the review process is shown in Fig 1. A total of 4,642 records were identified; 4,632 from databases and 10 from a search of references. Of these, 874 were duplicates and 3,594 were excluded after screening their abstract/title. We retrieved 174 papers for appraisal of which 132 were subsequently excluded (S3 Appendix) and 42 studies were included in this review [1960]. No unpublished studies were identified.

All of the studies addressed the question regarding clinician accuracy, and 17 studies included information that addressed the question about which clinicians were more accurate at prognosticating than others (Table 1). The participants of 25 (58%) studies had cancer, one (2%) study concerned participants with liver disease, and 17 (40%) studies contained both patients with cancer and non-cancer diagnoses. To assess reliability of the quality assessment, every second paper (alphabetically) included in the review was also scored for quality by the second reviewer, with moderate agreement, k = .6334, p < .001 [61]. The patient population, prognostic factor, outcome, and statistic domains were generally at low risk of bias across the studies. The clinician population and attrition domains had moderate levels of bias. The risk of bias due to confounding variables was moderate to high (S4 Appendix).

How accurate are clinician predictions of survival in palliative care patients?

Of the 42 studies included, 20 reported prognostic estimates using only a categorical approach [19, 20, 2527, 30, 34, 36, 37, 4042, 4547, 49, 55, 56, 58, 60], 16 reported only continuous estimates [21, 22, 24, 28, 29, 31, 33, 38, 39, 44, 48, 5053, 57] and 3 studies reported only probabilistic estimates [32, 43, 59]. Two studies used both categorical and continuous estimates [23, 54] and one study reported all three types of estimates [35].

Studies reporting categorical prognostic estimates.

The papers varied widely in regards to the number of prognostic categories and the boundaries for each category. Some studies reported clinicians’ predictions about whether patients would survive to a particular time point (e.g. greater or less than 4 weeks) and others consisted of multiple categories (e.g. “days”, “weeks”, “months” or “years”) (Table 2). In some studies clinicians were asked an open survival question (i.e. continuous), but the data were subsequently reported categorically as either “accurate” (which contained an upper and lower threshold for inclusion of the category, such as ±33%), “under estimate”, or “overestimate”.

thumbnail
Table 2. Summary of studies in which clinicians were asked to predict survival using defined categories (categorical studies).

https://doi.org/10.1371/journal.pone.0161407.t002

The accuracy of the categorical prognostic estimates in the 21 studies for which percentage accuracy could be calculated are presented in Fig 2 [19, 20, 23, 2527, 3437, 40, 41, 4547, 49, 5456, 58, 60]. Two papers could not be included in the forest plot because the relevant data was not available. Twomey, O’Leary, & O’Brien [30] reported that clinicians were just as likely to overestimate as to underestimate survival. Casarett, Farrington, & Craig et al [42] reported the c-statistic for the accuracy of predicting death between one and 10 days. They reported that the c-statistic varied between a minimum of 0.61 (14 day survival) to a maximum of 0.72 (7 day survival). A c-statistic of 0.7 or higher is generally considered acceptable evidence of ability to discriminate [62].

thumbnail
Fig 2. Summary data from studies in which clinicians provided categorical survival estimates (grouped by number of categories).

The data represented is the percentage of accurate estimates given out of the total number of estimates given. Note: The study by Gwilliam et al (2013) included doctor, nurse and MDT estimates. However, since the estimates were not independent of each other, only the MDT estimates have been presented here.

https://doi.org/10.1371/journal.pone.0161407.g002

The included studies reported 8,338 prognostic estimates. Fig 2 shows the variation in accuracy between the studies (range 23% to 78%). In one study [19] with only two prognostic categories, accuracy was reported as 78% (96% CI 75.8, 80.7); in contrast another study [20] also with only two prognostic categories, reported accuracy at 32% (95% CI 22.7, 42.3). Similarly with seven prognostic categories to choose from, one study [37] reported accuracy as 73% (95% CI 66.9, 78.2) and another study [35] reported accuracy as 34% (95% CI 31.8, 36.2). Possible reasons for this spread of results include variation in the number of pre-defined categories between the studies, the diverse nature of the clinical populations (age, diagnoses, gender mix), the characteristics of the clinicians, and the setting for the studies (hospice, community or hospital). Eight studies [19, 20, 23, 2527, 35, 47] reported estimates from more than one type of clinician, typically doctors and nurses. Thirteen papers reported the estimates of a single group of clinicians [34, 36, 37, 40, 41, 45, 46, 49, 5456, 58, 60]. Nine of these studies reported the accuracy of the medical profession [34, 37, 40, 49, 5456, 58, 60]; three studies reported the accuracy of a multidisciplinary team [36, 45, 46]; and one study [41] reported the accuracy of nurses. (S5 Appendix).

Studies reporting continuous prognostic estimates.

Table 3 shows data from 17 studies involving 4,511 continuous prognostic estimates [2124, 28, 29, 31, 33, 35, 38, 39, 44, 48, 50, 51, 54, 57]. As with the categorical data the results from these studies were very heterogeneous. The studies show that predicted median survival ranged from 14 to 219 days and actual median survival ranged from 10 to 126 days. The difference between median predicted and median actual survival ranged from an underestimate of 86 days to an overestimate of 93 days. In five of the studies the median difference showed an underestimate, while thirteen showed an overestimate.

thumbnail
Table 3. Predicted versus actual survival in those studies where clinicians were asked to provide a continuous temporal estimate of survival (continuous studies).

https://doi.org/10.1371/journal.pone.0161407.t003

Fig 3 shows the professional error score for each study. Two of the studies [35, 48] in Fig 3 appear twice as each study reported two separate prognostic estimates: from doctors and nurses in one paper [35] and upper and lower estimates in another [48].

thumbnail
Fig 3. Professional Error Score (PES) of clinicians’ estimates of survival those studies where clinicians were asked to provide a continuous temporal estimate of survival.

The black bar in this figure indicates the overall accuracy of the clinicians’ estimates (0 indicates perfect accuracy, positive values indicate over-estimates and negative values indicate under-estimates).

https://doi.org/10.1371/journal.pone.0161407.g003

Mackillop & Quirk [43] reported that doctors have an acceptable ability at predicting survival of three months (ROC value = 0.75, ±0.04 SE), but they are only slightly better than a random guess at 1 year (0.57, ±0.01 SE). Morita, Tsunoda, & Inoue et al [46] reported the results of two studies. The first study evaluated clinical predictions of survival by palliative care physicians and reported a moderate correlation between estimated and actual survival (r = 0.62). The second study evaluated clinical predictions of survival by palliative care physicians, with the aid of a prognostic tool, and reported a strong correlation between estimated and actual survival (r = 0.74).

Studies reporting probabilistic prognostic estimates.

Four papers used a probabilistic scale as a measure of accuracy [32, 35, 43, 59]. Hui, Kilgore, & Nguyen et al [35] asked doctors and nurses, “What is the approximate probability that this patient will be alive (0%–100%)?” for ≤24 hours, 48 hours, 1 week, 2 weeks, 1 month, 3 months, and 6 months. If a patient survived and the clinician had a survival percentage prediction of ≥70%, or the patient died within the time frame and the clinician had a survival prediction of ≤30%, they were considered correct estimates. Their results indicated that probabilistic prognostic estimates were more accurate than a continuous approach, for each time frame (p = < .001 for all paired comparisons). The other three [32, 43, 59] studies asked clinicians what was the percentage likelihood of six month survival and reported a ROC value between 0.74–0.78, which can be interpreted as demonstrating an acceptable level of accuracy.

Are any subsets of clinicians more “expert” at prognosticating than others?

In total, 17 studies were identified which addressed the issue of which group of clinicians are more accurate than others [1931, 35, 38, 47, 58].

Clinician characteristics.

Nine papers provided only minimal details about the clinicians (e.g. job title or specialty) [19, 23, 24, 2730, 38, 58]. Five papers reported the experience of the clinician; either in years, or narratively [20, 21, 25, 26, 31]. Bruera, Miller, & Kuehn et al [20] studied two clinicians who were described as “highly experienced”. Three papers provided more detailed characteristics about the clinicians [22, 35, 63] (Table 4).

thumbnail
Table 4. Summary of studies investigating differences in prognostic accuracy between clinical groups.

https://doi.org/10.1371/journal.pone.0161407.t004

Christakis & Lamont [22] reported clinicians’ individual characteristics: job title, self-rated optimism, experience, gender, board certification. They reported the doctor-patient relationship: how long they had known the patient, how frequently they had seen the patient, and the last time they saw the patient. They reported how many times they had referred someone to a hospice in the last year and how many patients they had met with a similar diagnosis. Gwilliam, Keeley & Todd et al [47] reported the following clinician characteristics: age; gender; length of time qualified; length of time working in palliative care how long they had known the patient and when they had last assessed them. Hui, Kilgore, & Nguyen et al [35] reported the clinicians’ age, gender, ethnicity, religion and years of experience (overall and within palliative care).

Overall differences in prognostic ability between clinicians

Overall, 7/17 (35%) studies found no difference in prognostic ability between different types of clinicians (however defined) [1921, 25, 26, 29, 58] (Table 4).

Six studies identified a difference between the prognostic accuracy of different clinicians. Gwilliam, Keeley & Todd et al [47] reported that a multidisciplinary estimate was more accurate than a nurse or doctor individually. They also reported that accuracy was not affected by gender, age, grade, experience, or length of time that the clinician had known the patient. However, nurses who had assessed a patient within the last 24 hours were more accurate than nurses who had not seen the patient within that time frame (p<0.01). Fairchild, Debenham, & Danielson, et al [23] compared the accuracy of doctors, radiation therapists, nurses, and allied health professionals. They reported that, overall, there was no difference between the prognostic accuracy of these groups, but radiation therapists were more accurate than allied health professionals. Twomey, O'Leary, & O'Brien[30] studied oncologists with varying levels of experience, social workers, and nurses. They reported that overall accuracy was below 50%, but nurses and junior doctors were more accurate than care assistants, nurse managers, and consultants. Heyse-Moore & Johnson-Bell [38] reported the accuracy of referrals to a hospice from hospital doctors and general practitioners (GPs). Although no comparisons were reported in the study, the results suggest that GPs were more accurate at predicting survival than the other groups. Llobera, Esteva, & Rifa et al [17] studied oncologists’, nurses’, and GPs’ estimates. They report that oncologists and nurses were more accurate than GPs. Forster & Lynn [29] studied oncologists of different grades, social workers, and nurses. They found that, whilst oncologists were more accurate than the other groups, accuracy overall was still optimistic.

Two studies found that the time frame of the prognosis can impact the accuracy. Hui, Kilgore, & Nguyen et al [38] found that nurses are better at predicting imminent death, whereas doctors are better at predicting three and six month survival. Oxenham & Cornbleet [48] reported the accuracy of a hospice team, consisting of a doctor, a sister, a staff nurse, a chaplain, and auxiliary staff. The results from this study suggest that doctors are more accurate than other groups when asked to provide a prognostic estimate at the initial assessment, but that auxiliary staff members are better at predicting when a patient’s death is imminent.

Two studies reported that experience can lessen prognostic errors. Christakis & Lamont [22] studied doctors’ estimates, documenting the characteristics of the doctors. The results suggested that overall, accuracy was low, but that length of experience may decrease the risk of errors (both over estimates and under estimates), as the more experienced doctors were less likely to make an error. Maltoni, Priovano, & Scarpi et al [31] reported that more experienced oncologists were more accurate at prognosticating.

Discussion

This systematic review identified 42 papers, spanning almost 30 years of research, and providing data on over 12,000 prognostic estimates. When clinicians were asked to provide a prognostic estimate from a pre-defined list of outcomes, accuracy varied from 23% to 78%. The clinical heterogeneity of the studies made it inadvisable to calculate an overall accuracy score. A previous systematic review [11] calculated an overall pooled accuracy score despite the high level of observed heterogeneity. Applying the same approach to our results would have indicated that clinicians over estimated survival by a factor of approximately two (44 days median predicted survival, 25 days median actual survival). However, for the reasons previously stated, this result should be viewed with caution. Although only recorded in four papers the evidence suggests that probabilistic estimates may be slightly more accurate than categorical or continuous temporal estimates of survival. There was no consistent evidence that one professional group or sub-group of clinicians was any more accurate than any other profession or sub-group. The level of experience of the clinician in some studies [31, 42] was seen as a factor that improves accuracy; however this was not replicated in all studies [20, 21, 25]. The time frame of the prognosis (e.g. prediction of imminent death versus prediction of death within 12 months) appeared to affect both the accuracy overall and the relative accuracy of different professionals [26, 35]. Some of the studies suggested that nurses and healthcare assistants are better at recognising imminent death than other professionals [28, 35]. Finally, two studies suggested that accuracy is better when the prognosis is made by a multidisciplinary team rather than by an individual clinician [25, 47].

Strengths and Limitations

This is the first systematic review which has analysed the accuracy of prognostic estimates of all diagnoses, according to the type of estimate (categorical, continuous or probabilistic), and the characteristics of the clinician (e.g. professional group, years of experience).

Cochrane recommends that ideally two independent reviewers should extract data from identified papers during a systematic review [61]. In this review, only one reviewer extracted data which could introduce bias to the results. However, we feel that the potential for bias that this introduced has been limited through the use of a standardised extraction table which specified which information, agreed by all authors, was to be extracted in order to address the review questions.

It was challenging to find a comprehensive search strategy to identify all relevant studies for inclusion in this review. In order to ensure that we identified as many relevant studies as possible, the specificity of our search strategy was relatively low and hence a large number of studies were initially identified. Even with a low specificity search strategy, ten of the studies included in this systematic review were only identified during the hand search of references, which raises the possibility that our search may have not identified all potential studies. We wanted to identify only those studies where clinicians specifically quantified the prognosis (in terms of duration or probability of survival). The Gold Standards Framework is an approach to optimising care for patients approaching the end of life and is widely used in general practice, care homes and hospitals [64]. As part of the GSF approach clinicians are encouraged to identify those patients about whom they would not be “surprised” if they died within the next year. Although this screening question is quite widely used for identification of patients approaching the end of life, we did not include any studies evaluating this approach in our review because it does not require clinicians to estimate how long a patient is expected to live, nor to gage the probability that they will die within the next year.

Due to the degree of clinical heterogeneity among included studies, it was not possible to conduct a meta-analysis to provide a pooled estimate of accuracy. Studies were clustered by the type of prognostic estimate that was obtained (categorical, continuous or probabilistic). However, even within these subgroups, categorical and continuous studies were still highly heterogeneous. Diverse outcome measures, missing data, and limited information on the demographics of the clinicians made the accuracy of categorical estimates and the question about which clinicians are better prognosticators difficult to address. Details about the clinicians being asked to provide a prognosis in the included studies were often limited. Additional information about clinicians (beyond simply reporting their profession) was only provided in 8/17 (47%) studies. This lack of information limited our analysis about the factors which distinguished more “expert” prognosticators from those less accomplished in this clinical skill.

The appraisal of the quality of each study was challenging. The method of appraising prognostic studies is currently under development by the Cochrane Prognosis Group. The QUIPS tool is suggested by the Cochrane group as a suitable risk of bias instrument; however some of the areas covered by the tool were not always relevant to this systematic review (e.g. the concept of attrition). The accuracy of survival estimates, particularly when considered by profession of clinician, was often a secondary analysis in the included papers rather than being the primary outcome. Any systematic review is only as good as its included studies. Our assessment of the quality of individual studies suggested moderate to high risk of bias due to confounding factors; however most of the domains assessed were rated as having low risk of bias.

Future implications of this review

Accurate prognoses are recognised as being of clinical importance for patients at all stages of the palliative care trajectory, from those recently referred to palliative care services [63] to those patients approaching the end of life [65]. Accuracy of categorical estimates in this systematic review ranged from 23% up to 78% and continuous estimates over-predicted actual survival by, potentially, a factor of two. This systematic review highlights the heterogeneous nature of studies of prognostic accuracy in palliative care. Future research, to potentially reduce the heterogeneity and increase accuracy, could be to incorporate a validated prognostic tool [53], using agreed “clinically relevant” prognostic categories. Examples of such tools include the Prognosis in Palliative Care predictor models[63] and the Palliative Prognostic score [66]. Alternatively, a recent review article [67], highlights that treatment plans and decisions can be made without such a weighted focus on survival estimates.

The Neuberger review into the use of the LCP [9] recommended that evidence-based education and competency-based training should be promoted to improve prognostic skills. However no clear guidance exists on how clinicians can be taught to perform this task better. There are currently no evidence-based education programmes to train clinicians how to become better prognosticators. Some studies suggest that more experienced or better qualified clinicians or different members of the MDT may be better than others at making prognostic predictions. Future research should try to identify how these clinicians have become better prognosticators so that evidence-based training can be developed for their less accurate colleagues.

Supporting Information

S2 Appendix. Search Strategy.

This is the strategy that was employed on the OVID platform and modified for other databases.

https://doi.org/10.1371/journal.pone.0161407.s002

(DOCX)

S3 Appendix. Breakdown of full study exclusion reasons.

https://doi.org/10.1371/journal.pone.0161407.s003

(XLSX)

S5 Appendix. Data used to complete the analysis of categorical estimates.

https://doi.org/10.1371/journal.pone.0161407.s005

(XLSX)

Acknowledgments

We’d like to thank Judith Scammel, Dr Paul Taylor, Dr Rebecca Jones, & Victoria Vickerstaff for their academic support during the completion of this systematic review. We’d also like to thank the authors of identified papers for their assistance and correspondence with their published work included in this review.

Author Contributions

  1. Conceptualization: NW FR PS.
  2. Data curation: NW FR PS PH AH.
  3. Formal analysis: NW FR PS.
  4. Funding acquisition: PS.
  5. Investigation: NW PS.
  6. Methodology: NW FR PS.
  7. Project administration: NW FR PS PH AH.
  8. Resources: NW FR PS.
  9. Supervision: PH AH PS.
  10. Validation: NW FR PS.
  11. Visualization: NW FR PS PH AH.
  12. Writing – original draft: NW FR PS PH AH.
  13. Writing – review & editing: NW FR PS PH AH.

References

  1. 1. Adams E, Boulton M, Watson E. The information needs of partners and family members of cancer patients: a systematic literature review. Patient education and counseling. 2009;77(2):179–86. pmid:19406609
  2. 2. Degner LF, Kristjanson LJ, Bowman D, Sloan JA, Carriere K, O'Neil J, et al. Information needs and decisional preferences in women with breast cancer. Jama. 1997;277(18):1485–92. pmid:9145723
  3. 3. Steinhauser KE, Christakis NA, Clipp EC, McNeilly M, Grambow S, Parker J, et al. Preparing for the end of life: preferences of patients, families, physicians, and other care providers. Journal of Pain and Symptom Management. 2001;22(3):727–37. pmid:11532586
  4. 4. Kirk P, Kirk I, Kristjanson LJ. What do patients receiving palliative care for cancer and their families want to be told? A Canadian and Australian qualitative study. BMJ. 2004;328(7452):1343. pmid:15151964
  5. 5. Kutner JS, Steiner JF, Corbett KK, Jahnigen DW, Barton PL. Information needs in terminal illness. Social science & medicine. 1999;48(10):1341–52.
  6. 6. Steinhauser KE, Christakis NA, Clipp EC, McNeilly M, McIntyre L, Tulsky JA. Factors considered important at the end of life by patients, family, physicians, and other care providers. Jama. 2000;284(19):2476–82. pmid:11074777
  7. 7. Glare PA, Sinclair CT. Palliative medicine review: prognostication. Journal of palliative medicine. 2008;11(1):84–103. pmid:18370898
  8. 8. Franks PJ, Salisbury C, Bosanquet N, Wilkinson EK, Kite S, Naysmith A, et al. The level of need for palliative care: a systematic review of the literature. Palliative medicine. 2000;14(2):93–104. pmid:10829143
  9. 9. Neuberger J, Guthrie C, Aaronovitch D. More care, less pathway: a review of the Liverpool Care Pathway. Department of Health, Crown Copyright. 2013.
  10. 10. Chow E, Harth T, Hruby G, Finkelstein J, Wu J, Danjoux C. How accurate are physicians' clinical predictions of survival and the available prognostic tools in estimating survival times in terminally ill cancer patients? A systematic review. 2001;13:209–18.
  11. 11. Glare P, Virik K, Jones M, Hudson M, Eychmuller S, Simes J, et al. A systematic review of physicians' survival predictions in terminally ill cancer patients. BMJ. 2003;327(7408):195. pmid:12881260
  12. 12. Zhou M, Holden L, Lao N, Lam H, Zeng L, Chow E. Accuracy of Clinicians’ Prediction of Survival and Prognostic Factors Indicative of Survival: a Systematic Review. Hong Kong Journal of Radiology. 2013;16(3):168–82.
  13. 13. Coventry PA, Grande GE, Richards DA, Todd CJ. Prediction of appropriate timing of palliative care for older adults with non-malignant life-threatening disease: a systematic review. Age and ageing. 2005;34(3):218–27. pmid:15863407
  14. 14. Cochrane Methods Prognosis. Available: http://methods.cochrane.org/prognosis.
  15. 15. Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ. 2001;323(7306):224–8. pmid:11473921
  16. 16. Tavares FA. Comparing the accuracy of the D-PaP and the PaP score in patients with diagnoses other than cancer: Palliative Medicine. Conference: 8th World Research Congress of the European Association for Palliative Care, EAPC 2014 Lleida Spain. Conference Start: 20140605 Conference End: 20140607. Conference Publication: (var.pagings). 28 (6) (pp 667–668), 2014. Date of Publication: June 2014.; 2014.
  17. 17. Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. BMJ. 2009;339.
  18. 18. Hayden JA, van der Windt DA, Cartwright JL, CÃ P, Bombardier C. Assessing bias in studies of prognostic factors. Annals of Internal Medicine. 2013;158(4):280–6. pmid:23420236
  19. 19. Addington-Hall JM, MacDonald LD, Anderson HR. Can the Spitzer Quality of Life Index help to reduce prognostic uncertainty in terminal care? British journal of cancer. 1990;62(4):695–9. pmid:2223593
  20. 20. Bruera E, Miller MJ, Kuehn N, MacEachern T, Hanson J. Estimate of survival of patients admitted to a Palliative Care Unit: A prospective study. Journal of Pain and Symptom Management. 1992;7(2):82–6. pmid:1573289
  21. 21. Chow E, Davis L, Panzarella T, Hayter C, Szumacher E, Loblaw A, et al. Accuracy of survival prediction by palliative radiation oncologists. International Journal of Radiation Oncology Biology Physics. 2005;61(3):870–3.
  22. 22. Christakis NA, Lamont EB. Extent and determinants of error in doctors' prognoses in terminally ill patients: prospective cohort study. BMJ (Clinical research ed). 2000;320(7233):469–72.
  23. 23. Fairchild A, Debenham B, Danielson B, Huang F, Ghosh S. Comparative multidisciplinary prediction of survival in patients with advanced cancer. Supportive care in cancer: official journal of the Multinational Association of Supportive Care in Cancer. 2014;22(3):611–7.
  24. 24. Forster LE, Lynn J. Predicting life span for applicants to inpatient hospice. 1988;148:2540–3. pmid:3196122
  25. 25. Gripp S, Moeller S, Bölke E, Schmitt G, Matuschek C, Asgari S, et al. Survival prediction in terminally ill cancer patients by clinical estimates, laboratory tests, and self-rated anxiety and depression. 2007;25:3313–20. pmid:17664480
  26. 26. Hølmebakk T, Solbakken a, Mala T, Nesbakken a. Clinical prediction of survival by surgeons for patients with incurable abdominal malignancy. European Journal of Surgical Oncology. 2011;37(7):571–5. pmid:21444178
  27. 27. Llobera J, Esteva M, Rifa J, Benito E, Terrasa J, Rojas C, et al. Terminal cancer: duration and prediction of survival time. European journal of cancer. 2000;36(16):2036–43. pmid:11044639
  28. 28. Oxenham D, Cornbleet Ma. Accuracy of prediction of survival by different professional groups in a hospice. 1998;12:117–8. pmid:9616448
  29. 29. Parkes CM. Accuracy of predictions of survival in later stages of cancer. BMJ. 1972;2(5804):29–31. pmid:4111472
  30. 30. Twomey F, O'Leary N, O'Brien T. Prediction of patient survival by healthcare professionals in a specialist palliative care inpatient unit: a prospective study. The American journal of hospice & palliative care. 2008;25(2):139–45.
  31. 31. Maltoni M, Nanni O, Derni S, Innocenti MP, Fabbri L, Riva N, et al. Clinical prediction of survival is more accurate than the Karnofsky performance status in estimating life span of terminally ill cancer patients. European journal of cancer. 1994;30A(6):764–6. pmid:7917534
  32. 32. Knaus WA, Harrell FE, Lynn J, Goldman L, Phillips RS, Connors AF, et al. The SUPPORT prognostic model: Objective estimates of survival for seriously ill hospitalized adults. Annals of Internal Medicine. 1995;122(3):191–203.
  33. 33. Evans C, McCarthy M. Prognostic uncertainty in terminal care: can the Karnofsky index help? Lancet. 1985;1(8439):1204–6. pmid:2860399
  34. 34. Glare P, Virik K. Independent prospective validation of the PaP score in terminally ill patients referred to a hospital-based palliative medicine consultation service. Journal of Pain and Symptom Management. 2001;22(5):891–8. pmid:11728792
  35. 35. Hui D, Kilgore K, Nguyen L, Hall S, Fajardo J, Cox-Miller TP, et al. The Accuracy of Probabilistic Versus Temporal Clinician Prediction of Survival for Patients with Advanced Cancer: A Preliminary Report. 2011;16:1642–8. pmid:21976316
  36. 36. Selby D, Chakraborty A, Lilien T, Stacey E, Zhang L, Myers J. Clinician accuracy when estimating survival duration: The role of the patient's performance status and time-based prognostic categories. 2011;42:578–88. pmid:21565461
  37. 37. Thomas JM, O'Leary JR, Fried TR. Understanding their options: Determinants of hospice discussion for older persons with advanced illness. 2009;24:923–8. pmid:19506972
  38. 38. Heyse-Moore L, Johnson-Bell V. Can doctors accurately predict the life expectancy of patients with terminal cancer? Palliative medicine. 1987;1(2):165–6.
  39. 39. Maltoni M, Priovano M, Scarpi E, Marinari M, Indelli M, Arnoldi E, et al. Prediciton of Survival of Patients Terminally Ill with Cancer. 1995;75:2613–22.
  40. 40. Brandt HE, Ooms ME, Ribbe MW, van der Wal G, Deliens L. Predicted Survival vs. Actual Survival in Terminally Ill Noncancer Patients in Dutch Nursing Homes. 2006;32:560–6. pmid:17157758
  41. 41. Buchan JEF. Nurses' estimations of patients' prognoses in the last dats of life. International Journal of Palliative Nursing. 1995;1(1):12–6.
  42. 42. Casarett DJ, Farrington S, Craig T, Slattery J, Harrold J, Oldanie B, et al. The Art versus Science of Predicting Prognosis: Can a Prognostic Index Predict Short-Term Mortality Better than Experienced Nurses Do? 2012;15:703–8. pmid:22583382
  43. 43. Cooper GS, Bellamy P, Dawson NV, Desbiens N, Fulkerson W.J Jr, Goldman L, et al. A prognostic model for patients with end-stage liver disease. 1997;113:1278–88. pmid:9322523
  44. 44. Faris M. Clinical estimation of survival and impact of other prognostic factors on terminally ill cancer patients in Oman. 2003;11:30–4. pmid:12527951
  45. 45. Fromme EK, Smith MD, Bascom PB, Kenworthy-Heinige T, Lyons KS, Tolle SW. Incorporating routine survival prediction in a U.S. hospital-based palliative care service. Journal of palliative medicine. 2010;13(12):1439–44. pmid:21128821
  46. 46. Glare PA, Eychmueller S, McMahon P. Diagnostic accuracy of the palliative prognostic score in hospitalized patients with advanced cancer. 2004;22:4823–8. pmid:15570085
  47. 47. Gwilliam B, Keeley V, Todd C, Roberts C, Gittins M, Kelly L, et al. Prognosticating in patients with advanced cancer-observational study comparing the accuracy of clinicians' and patients' estimates of survival. Annals of Oncology. 2013;24(2):482–8. pmid:23028038
  48. 48. Higginson IJ, Costantini M. Accuracy of prognosis estimates by four palliative care teams: a prospective cohort study. BMC palliative care. 2002;1:1–. pmid:11876829
  49. 49. Kao SCH, Butow P, Bray V, Clarke SJ, Vardy J. Patient and oncologist estimates of survival in advanced cancer patients. Psycho-Oncology. 2011;20(2):213–8. pmid:20878829
  50. 50. Lam PT. Accuracy of clinical prediction of survival in a palliative care unit. Progress in Palliative Care. 2008;16(3):113–7.
  51. 51. Lamont EB, Christakis NA. Prognostic disclosure to patients with cancer near the end of life. Annals of internal medicine. 2001.
  52. 52. Mackillop WJ, Quirt CF. Measuring the accuracy of prognostic judgments in oncology. Journal of Clinical Epidemiology. 1997;50(1):21–9. pmid:9048687
  53. 53. Morita T, Tsunoda J, Inoue S, Chihara S. Improved accuracy of physicians' survival prediction for terminally ill cancer patients using the Palliative Prognostic Index. 2001;15:419–24. pmid:11591094
  54. 54. Muers MF, Shevlin P, Brown J. Prognosis in lung cancer: physicians' opinions compared with outcome and a predictive model. Thorax. 1996;51(9):894–902. pmid:8984699
  55. 55. Shah S, Blanchard M, Tookman A, Jones L, Blizard R, King M. Estimating needs in life threatening illness: a feasibility study to assess the views of patients and doctors. Palliative medicine. 2006;20(3):205–10. pmid:16764225
  56. 56. Stiel S, Bertram L, Neuhaus S, Nauck F, Ostgathe C, Elsner F, et al. Evaluation and comparison of two prognostic scores and the physicians' estimate of survival in terminally ill patients. 2010;18:43–9. pmid:19381693
  57. 57. Taniyama TK, Hashimoto K, Katsumata N, Hirakawa a, Yonemori K, Yunokawa M, et al. Can oncologists predict survival for patients with progressive disease after standard chemotherapies? Current Oncology. 2014;21(2):84–90. pmid:24764697
  58. 58. Viganò A, Dorgan M, Bruera E, Suarez-Almazor ME. The relative accuracy of the clinical estimation of the duration of life for patients with end of life cancer. Cancer. 1999;86(1):170–6. pmid:10391577
  59. 59. Weeks JC, Cook EF, O'Day SJ, Peterson LM, Wenger N, Reding D, et al. Relationship between cancer patients' predictions of prognosis and their treatment preferences. JAMA. 1998;279(21):1709–14. pmid:9624023
  60. 60. Zibelman M, Xiang Q, Muchka S, Nickoloff S, Marks S. Assessing Prognostic Documentation and Accuracy Among Palliative Care Clinicians. Journal of palliative medicine. 2014;17(5):1–6.
  61. 61. Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions: Wiley Online Library; 2008.
  62. 62. Hosmer DW, Lemeshow S. Assessing the Fit of the Model. Applied Logistic Regression: John Wiley & Sons, Inc.; 2005. p. 143–202.
  63. 63. Gwilliam B, Keeley V, Todd C, Gittins M, Roberts C, Kelly L, et al. Development of Prognosis in Palliative care Study (PiPS) predictor models to improve prognostication in advanced cancer: prospective cohort study. 2011;343:d4920–d. pmid:21868477
  64. 64. Prognostic Indicator Guidance (PIG) The Gold Standards Framework Centre In End of Life Care CIC. 2011. Thomas Kea. [04/05/2016]; 4th:[Available: http://www.goldstandardsframework.org.uk/cd-content/uploads/files/General%20Files/Prognostic%20Indicator%20Guidance%20October%202011.pdf.
  65. 65. Leadership Alliance for the Care of Dying People. 2014. One chance to get it right: Improving people's experience of care in the last few days and hours of life. Available: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/323188/One_chance_to_get_it_right.pdf.
  66. 66. Pirovano M, Maltoni M, Nanni O, Marinari M, Indelli M, Zaninetta G, et al. A New Palliative Prognostic Score: A First Step for the Staging of Terminally Ill Cancer Patients. Journal of Pain and Symptom Management. 1999;17(4):231–9. pmid:10203875
  67. 67. Chiu N, Chiu L, Lutz S, Zhang N, Lechner B, Pulenzas N, et al. Incorporation of life expectancy estimates in the treatment of palliative care patients receiving radiotherapy: treatment approaches in light of incomplete prognostic models. Annals of Palliative Medicine. 2015;4(3):162–8. pmid:26231813