Weiyi Xie
Robyn Butcher
Key Messages
Why Is This an Issue?
End-of-life care provides support for patients and their families during the last stage of life. End-of-life conversations aim to help people better understand their disease prognosis and expected survival, enabling them to make informed decisions regarding end-of-life care.
Palliative care focuses on relieving symptoms and improving quality of life for patients with serious or life-threatening diseases. Approximately 89% of patients with life-limiting diseases, such as cancer, can benefit from palliative care. However, not all patients receive it in a timely manner.
Due in part to prognostic uncertainty and optimism bias, end-of-life planning conversations and palliative care decisions do not occur early enough to have maximum benefit. Interventions that aim to prompt or help identify those patients who can benefit from palliative and/or end-of-life planning could improve the quality of care.
What Is the Technology?
An artificial intelligence (AI)–based “nudge” is a decision-making support tool that uses prompts and alerts to aid clinicians in deciding whether and when to discuss end-of-life planning with patients.
The nudge sends alerts and/or reminders to clinicians to prompt end-of-life conversations with patients who are at high risk of short-term mortality. These patients are identified by machine learning mortality prediction algorithms incorporated in the electronic health record (EHR) system.
Two AI-based nudges designed for patients with cancer were identified. Both tools were developed and internally validated in the US.
What Is the Potential Impact?
AI-based nudges have the potential to increase the number of end-of-life planning conversations between clinicians and patients as well as the number of referrals to end-of life services.
Implementing the nudges into clinical workflows could also help clinicians more easily identify patients with palliative care needs.
What Else Do We Need to Know?
No AI-based nudges have been approved for use in Canada at the time of this writing nor have there been validation studies using Canadian data.
As with many AI algorithms, there is uncertainty about the validity and generalizability of the mortality predictive algorithms used in the nudges.
The acceptance of AI-based nudges by clinicians is unclear due to varying clinician attitudes and experiences with nudges and because we did not identify any studies that reported the experience of AI-based nudges from the patient perspective.
An AI-based behavioural intervention, referred to as a “nudge,” uses alerts and/or reminders to prompt clinicians to initiate end-of-life conversations with patients under their care who are at high risk of short-term (such as 30-day1-3 or 180-day) mortality.4-6 These patients are identified by machine learning mortality prediction tools incorporated in the EHR system.1,2,4-7 Through end-of-life conversations, patients can better understand their disease prognosis and expected survival, enabling them to make informed decisions regarding future treatment options.8 For example, they could decide whether to choose disease-directed treatment or palliative care, which focuses on relieving symptoms and improving quality of life for patients with serious or life-threatening diseases.9
Two machine learning AI-based nudges, 1 of which has been commercialized, have been developed to assist oncologists in the US to decide whether and when to discuss palliative care and end-of-life planning with patients.1-6 Machine learning is a domain of AI that trains computer algorithms to provide accurate predictions based on identified patterns from historical data.10,11 Clinicians often overestimate patient survival.12-17 Machine learning–based tools use data from EHRs and/or other sources to provide a more precise estimate of short-term mortality.18 Moreover, machine learning AI-based nudges can create a list of patients at high risk of 30-day1-3 or 180-day mortality,4-6 with thresholds tailored to clinician capacity for end-of-life planning conversations. For example, a nudge developed by Manz et al.4 uses structured electronic record data to identify the 10% of patients at the highest risk of 180-day mortality.
AI-based nudges generally offer weekly notifications within a secure platform, identifying patients at high risk for short-term mortality1,2,4-7 and supporting the identification of those in need of advance care planning.
For example, the nudge designed by Manz et al.4 delivers prompts through a secure dashboard, which encourage oncologists to initiate “serious illness conversations” (SICs) through 3 approaches:
a weekly personalized list of up to 6 highest-risk patients for 180-day mortality, including patient identifiers, the health professional and date of any prior SICs, and a default checkbox for opting out of reminder texts5-7
an opt-out message sent on the patient’s appointment day to prompt clinicians to consider initiating the SIC5-7
a weekly email regarding the number of SICs clinicians conducted in the preceding 4 weeks and a peer comparison message tailored to clinicians who performed fewer or more than 8 SICs or ranked among the top 10 performers of SICs during the previous 4 weeks.5-7
AI-based nudges developed for oncologists1-6 may improve early access to palliative care for patients with cancer. In 2022, it was estimated that there were 85,100 cancer-related deaths in Canada.19 According to a 2023 report from the Canadian Institute for Health Information (CIHI), 89% of people with life-limiting diseases, such as cancer and kidney diseases, can benefit from palliative care.20 In 2021 to 2022, 58% of people who died (89,000 people) in Canada were described as palliative and had received some form of palliative care.20
However, CIHI noted that not everyone in Canada receives palliative care early enough.20 It reported that, regardless of care setting, half of patients died within 22 days after being identified as palliative.20 Half of patients receiving care in hospital with identified palliative care needs lived 11 days or fewer following the identification.20 People with cancer often receive aggressive treatments near the end of life.21,22 These patients may experience severe physical and psychosocial symptoms before death. Palliative care can reduce these symptoms and enhance patients’ quality of life.20
AI-based nudges have the potential to help clinicians identify patients who could benefit from palliative and end-of-life care planning and facilitate earlier conversations about their treatment goals and end-of-life preferences.
Neither of the 2 nudges1-6 developed in the US have been approved for use in Canada.
The nudge developed by Manz et al.4 has not been commercialized, whereas another nudge1,2 has been commercialized for, and is available in, the US market.
Although no specific cost information is available, the price of an AI-based tool may include tool usage, training, and ongoing subscriptions for system updates and user support.23 Implementation training could help clinicians better interpret prediction results, address disagreements with AI predictions, and maintain responsiveness as the volume of system notification grows. Further, evidence suggests that clinicians often feel unprepared to initiate SIC conversations,24 highlighting training is needed to develop essential communication skills for these difficult conversations. For health care services without an EHR system, additional costs include implementing, maintaining, and training for the EHR software.
In Canada, decisions to initiate palliative care and/or end-of-life care are made by assessing the patient’s condition using tools such as the Palliative Performance Scale.20,25 However, many patients are not eligible until they approach the end of their lives.20
The Ontario Palliative Care Network recommends the Hospital One-year Mortality Risk (HOMR) tool for early identification of palliative care needs across different care settings.26 HOMR uses statistical methods to estimate mortality risk and is not AI enhanced. Refer to the Related Developments section for more details.
The prompt designed by Manz et al.4 was prospectively validated using new data from the original EHR systems, whereas the AI algorithm of the commercial nudge was validated within the same practice where it was developed.2 Both tools lack external validation with structurally different data from other health systems, so their performance in different populations is unknown. A summary of prediction performance of the 2 tools is provided in Table 1. In general, AI algorithms of both nudges demonstrated:
good overall performance in discriminating patients with cancer who had high risk or low risk of 30-day2 or 180-day mortality,4 with both receiver operating characteristic curves exceeding 0.82,4
accurate identification of patients at low risk of 30-day2 or 180-day mortality,4 as indicated by specificities and negative predictive values all exceeding 0.952,4
limited ability to identify high-risk patients, with sensitivity values dropping below 0.3, showing that they can capture fewer than 30% of high-risk patients.2,4
In addition, Manz et al.4 had a positive predictive value of 0.45. This means that for patients identified by the algorithm as having a high risk of 180-day mortality, there is a 45% chance of passing away within 180 days. The positive predictive value of the commercial nudge was unreported,2 suggesting signs of selective reporting in the validation study.
A stepped-wedge cluster randomized trial4-7 compared the nudge developed by Manz et al.4 with usual care in patients with cancer. The study involved 20,506 patients and 41,021 patient encounters, including 5,520 (13.5%) high-risk patient encounters.5
For all patient encounters, the study5 found:
a significant increase in SIC rates compared to the control period (4.4% vs. 1.3%)
a significant decrease in end-of-life systemic therapy compared to the control period (7.5% vs. 10.4%)
no impact on hospice enrolment or length of stay, inpatient death, or end-of-life intensive care unit use.
Among high-risk patient encounters, the study5 found:
a significant increase in SIC rates compared to the control period (13.5% vs. 3.4%).
A real-world before-after study1 reported that the commercial AI-based nudge significantly increased palliative care consults and hospice referrals for patients with cancer.1 In high- or medium-risk patients, the study1 found:
a nearly 2-fold increase in the average number of palliative care consults (after controlling for a 6 month adaptation period)
a nearly 13-fold increase in hospice referrals compared to baseline.
Both AI nudges demonstrated sensitivity less than 0.3,2,4 indicating they can correctly identify fewer than 30% of high-risk patients for mortality. Both nudges had a specificity exceeding 0.95,2,4 which means they can accurately identify more than 95% of patients without high mortality risk. The high specificity values suggest a potential trade-off between specificity and sensitivity. Sensitivity is generally prioritized for rare event prediction,18,27 such as mortality prediction.28 For example, some researchers preferred high sensitivity over specificity when designing a machine learning mortality predictive model for older adults.28 One systematic review18 noted that the low sensitivity of the nudge designed by Manz et al.4 could lead to its poor performance in identifying patients at high mortality risk. Nevertheless, the emphasis on sensitivity or specificity depends on the models’ purposes. In a prediction model designed to identify patients at high risk of short-term mortality, researchers may prioritize high specificity to minimize the misclassification of low-risk patients as high risk. This is likely particularly important for large clinical practices, where a high number of false-positives would either require clinicians to spend considerable time reviewing patient records and speaking with people who are not at high mortality risk or lead to the nudge being ignored due to the time required for review.
Manz et al. also employed a 40% threshold for high-risk patient identification during validation.4 This meant that the nudge would flag the top 40% of patients with the greatest risk of 180-day mortality as high risk, while the other patients were classified as low risk.4 But in real-world applications, researchers used a 10% threshold without assessing the nudge’s performance at this threshold.5-7 Consequently, it is unknown whether the nudge could predict mortality with a 10% threshold as precisely as it did using a 40% threshold.
There is uncertainty regarding the generalizability of AI-based nudges due to the lack of external validation. Researchers have raised concerns about potential data bias in algorithm training datasets,29,30 which influences the performance of AI models and may cause health inequities. For example, EHR-based AI models have demonstrated decreased performance and calibration across various geographic locations and over time.31 AI models also showed poor performance in historically underrepresented groups,31 potentially reinforcing existing inequities.
External validation using data from other health systems can test whether the algorithms have site bias or centre bias,32 which may result from variations in treatment protocols. It can also detect sampling bias and whether the models’ predictive performance changes when applied to new cohorts with different demographics.32 However, current AI-based nudges lack external validation. Therefore, it is unclear whether they can predict mortality as accurately in Canadian health care systems as they did in the development setting. For example, although the nudge developed by Manz et al.4 was prospectively validated using new data in the original system to detect algorithm overfitting to the development dataset, it cannot evaluate the nudge’s performance in other EHR systems. Therefore, this prospective internal validation cannot guarantee safe application of the model in other populations.
The effect of the AI-based nudges in clinical practice can be influenced by clinician acceptance. Evidence showed that clinicians varied in their attitudes toward machine learning–based clinical support tools.27
Regarding using AI-based nudges in routine care of patients with cancer, some clinicians believed it could aid in validating their own prognostic estimates and prompt conversations about treatments patients prefer to receive near the end of their lives.33
However, other clinicians have concerns about algorithm accuracy, over-reliance on algorithm predictions, and the ethical implications related to disclosure of algorithm predictions.33
In addition, clinician response rates to the alerts can affect the effectiveness of these nudges because clinicians are responsible for making clinical decisions, with the nudges serving as supportive tools. Behavioural studies revealed that, over time, the impact of EHR alerts decreased due to alert fatigue.34,35 Health care providers will pay less attention to alerts or reminders when responding to them requires a lot of time or a lot of effort.36 In a secondary analysis of a randomized controlled trial, the nudge developed by Manz et al.4 was applied in clinical practice and found that specialists with low patient volumes were more responsive than general oncologists and specialists with higher patient volumes.37 Therefore, low response rates from clinicians due to limited capacity or alert fatigue may be an important barrier to implementation, especially for nudges sending multiple prompts.
HOMR is an EHR electronic medical record–integrated application that automatically calculates 12-month mortality risk for newly admitted patients.38,39 Based on multivariable binomial logistic regression modelling, it predicts mortality risk using patient demographic variables and health administrative data.39 HOMR was internally validated using population-based health administrative databases in Ontario39 and externally validated in adult, nonpsychiatric, inpatient acute care settings in Ontario and Alberta as well as Boston, Massachusetts.38 HOMR can notify the admitting medical team of high-risk patients using an electronic sign-out tool within the EHR.40 It can also send email alerts to the admitting physician for consideration of palliative and end-of-life care needs.40 The Ontario Palliative Care Network recommends using HOMR in acute care settings.26 Some hospitals in Ontario have implemented HOMR either in practice26,41 or within clinical studies.42 In addition, the latest version, HOMR Now!, calculates risk immediately upon patient admission, which was internally validated using data from a tertiary-care teaching hospital in Ontario.43
Researchers at Washington University created and internally validated an AI-based nudge for the general inpatient population in a community setting.44 It notified physicians when a patient was identified as having a high risk of 30-day mortality to prompt the discussion about care goals before discharge.44 The AI model can predict short-term mortality or hospice outcomes on the second day of a patient’s admission.44 However, details about the delivery method and frequency of these alerts were unknown.
An AI-based behavioural nudge triggering palliative care intervention for hospitalized patients was developed and internally validated in the US.45 Unlike nudges using mortality as a proxy for potential palliative care needs, this tool identified patients at low risk of short-term mortality who may still benefit from palliative care.45 It was developed by learning the associations between variables in the electronic medical record and palliative care consultation.45 The nudge was delivered through a clinical Control Tower, where Control Tower Operators and palliative care service clinicians evaluated potential palliative care needs of patients flagged by the nudge.46
CADTH released a 2022 report introducing clinical applications of AI.47 The report included developments and application of predictive AI models that can estimate disease progression, patient outcomes, and overall survival across various medical conditions.47
AI-based behavioural nudges have the potential to assist clinicians in identifying high-risk patients for short-term mortality and initiating end-of-life planning conversations, thus improving timely referrals to palliative care or hospice services. Although there are nudges for patients with cancer that have been tested in real-world settings, the predictive performance and generalizability are uncertain due to the lack of external validation. In addition, the acceptance of AI prognostic prediction tools is unclear due to varying clinician attitudes and unknown patient perspectives. More validation studies are required to confirm the robustness and generalizability of machine learning AI models in various populations.
More real-world research is needed to investigate clinician and patient acceptance of these nudges.
Additional research that examines the use of AI-based nudges for patients with nonmalignant diseases and those who have reduced access to palliative care compared with patients with cancer would be helpful to understand whether the AI-based tools are helpful for conditions other than cancer.48-51 A 2023 report from CIHI indicated that among all patients with life-limiting illness in Canada, individuals with end-stage cancer were 3 times more likely to receive palliative care than those with other conditions.20 Developing AI-based nudges and/or other interventions for patients with nonmalignant diseases may facilitate equitable access to palliative care services.
1.Gajra A, Zettler ME, Miller KA, et al. Impact of augmented intelligence on utilization of palliative care services in a real-world oncology setting. JCO Oncol Pract. 2022;18(1):e80-e88. PubMed
2.Gajra A, Zettler ME, Miller KA, et al. Augmented intelligence to predict 30-day mortality in patients with cancer. Future Oncology. 2021;17(29):3797-3807. PubMed
3.Pharmacy Times. Augmented intelligence tool helps ensure timely referrals to palliative care for patients with advanced cancer. 2022; https://www.pharmacytimes.com/view/augmented-intelligence-tool-helps-ensure-timely-referrals-to-palliative-care-for-patients-with-advanced-cancer. Accessed 2023 Sep 26.
4.Manz CR, Chen J, Liu M, et al. Validation of a machine learning algorithm to predict 180-day mortality for outpatients with cancer. JAMA Oncol. 2020;6(11):1723-1730. PubMed
5.Manz CR, Zhang Y, Chen K, et al. Long-term effect of machine learning-triggered behavioral nudges on serious illness conversations and end-of-life outcomes among patients with cancer: a randomized clinical trial. JAMA Oncol. 2023;9(3):414-418. PubMed
6.Manz CR, Parikh RB, Evans CN, et al. Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: design and methods for a stepped-wedge cluster randomized controlled trial. Contemp Clin Trials. 2020;90:105951. PubMed
7.Manz CR, Parikh RB, Small DS, et al. Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol. 2020;6(12):e204759. PubMed
8.Goswami P. Impact of advance care planning and end-of-life conversations on patients with cancer: an integrative review of literature. J Nurs Scholarsh. 2023;55(1):272-290. PubMed
9.Weeks JC, Cook EF, O'Day SJ, et al. Relationship between cancer patients' predictions of prognosis and their treatment preferences. JAMA. 1998;279(21):1709-1714. PubMed
10.Chartrand G, Cheng PM, Vorontsov E, et al. Deep learning: a primer for radiologists. Radiographics. 2017;37(7):2113-2131. PubMed
11.Sidey-Gibbons JAM, Sidey-Gibbons CJ. Machine learning in medicine: a practical introduction. BMC Med Res Methodol. 2019;19(1):64. PubMed
12.Chow E, Davis L, Panzarella T, et al. Accuracy of survival prediction by palliative radiation oncologists. Int J Radiat Oncol Biol Phys. 2005;61(3):870-873. PubMed
13.Fairchild A, Debenham B, Danielson B, Huang F, Ghosh S. Comparative multidisciplinary prediction of survival in patients with advanced cancer. Support Care Cancer. 2014;22(3):611-617. PubMed
14.Kondziolka D, Parry PV, Lunsford LD, et al. The accuracy of predicting survival in individual patients with cancer. J Neurosurg. 2014;120(1):24-30. PubMed
15.Barnes EA, Chow E, Tsao MN, et al. Physician expectations of treatment outcomes for patients with brain metastases referred for whole brain radiotherapy. Int J Radiat Oncol Biol Phys. 2010;76(1):187-192. PubMed
16.Faris M. Clinical estimation of survival and impact of other prognostic factors on terminally ill cancer patients in Oman. Support Care Cancer. 2003;11(1):30-34. PubMed
17.Perez-Cruz PE, Dos Santos R, Silva TB, et al. Longitudinal temporal and probabilistic prediction of survival in a cohort of patients with advanced cancer. J Pain Symptom Manage. 2014;48(5):875-882. PubMed
18.Lu SC, Xu C, Nguyen CH, Geng Y, Pfob A, Sidey-Gibbons C. Machine learning-based short-term mortality prediction models for patients with cancer using electronic health record data: systematic review and critical appraisal. JMIR Med Inform. 2022;10(3):e33182. PubMed
19.Statistics Canada. New cancer estimates for 2022. 2022; https://www.statcan.gc.ca/o1/en/plus/1181-new-cancer-estimates-2022. Accessed 2023 Sep 26.
20.Access to palliative care in Canada, 2023. Ottawa (ON): Canadian Institute for Health Information; 2023: https://www.cihi.ca/sites/default/files/document/access-to-palliative-care-in-canada-2023-report-en.pdf. Accessed 2023 Sep 26.
21.Bylicki O, Didier M, Riviere F, Margery J, Grassin F, Chouaid C. Lung cancer and end-of-life care: a systematic review and thematic synthesis of aggressive inpatient care. BMJ Support Palliat Care. 2019;9(4):413-424. PubMed
22.Jang TK, Kim DY, Lee SW, et al. Trends in treatment during the last stages of life in end-stage gynecologic cancer patients who received active palliative chemotherapy: a comparative analysis of 10-year data in a single institution. BMC Palliat Care. 2018;17(1):99. PubMed
23.Brett K, Severn M. Facial analysis technology for pain detection: a potentially useful tool for people living with dementia. Can J Health Technol. 2023;3(7). https://canjhealthtechnol.ca/index.php/cjht/article/view/EN0047. Accessed 2023 Nov 08. PubMed
24.Fulmer T, Escobedo M, Berman A, Koren MJ, Hernández S, Hult A. Physicians' views on advance care planning and end-of-life care conversations. J Am Geriatr Soc. 2018;66(6):1201-1205. PubMed
25.BC Emergency Health Services. Palliative Performance Scale PPS. 2021; https://handbook.bcehs.ca/clinical-resources/clinical-scores/palliative-performance-scale-pps/. Accessed 2023 Sep 26.
26.Tools to support earlier identification for palliative care. Toronto (ON): Ontario Palliative Care Network; 2019: https://www.ontariopalliativecarenetwork.ca/sites/opcn/files/2021-01/OPCNToolsToSupportEarlierIdentificationForPC.pdf. Accessed 2023 Nov 22.
27.Hendrix N, Hauber B, Lee CI, Bansal A, Veenstra DL. Artificial intelligence in breast cancer screening: primary care provider preferences. J Am Med Inform Assoc. 2021;28(6):1117-1124. PubMed
28.Krasowski A, Krois J, Kuhlmey A, Meyer-Lueckel H, Schwendicke F. Predicting mortality in the very old: a machine learning analysis on claims data. Sci Rep. 2022;12(1):17464. PubMed
29.Artificial intelligence (AI) in healthcare and research. London (GB): Nuffield Council on Bioethics; 2018: https://www.nuffieldbioethics.org/wp-content/uploads/Artificial-Intelligence-AI-in-healthcare-and-research.pdf. Accessed 2023 Sep 26.
30.Forsting M. Machine learning will change medicine. J Nucl Med. 2017;58(3):357-358. PubMed
31.Lu J, Sattler A, Wang S, et al. Considerations in the reliability and fairness audits of predictive models for advance care planning. medRxiv. 2022;12.
32.Kernbach JM, Staartjes VE. Foundations of machine learning-based clinical prediction modeling: part II-generalization and overfitting. Acta Neurochir Suppl. 2022;134:15-21. PubMed
33.Parikh RB, Manz CR, Nelson MN, et al. Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study. Support Care Cancer. 2022;30(5):4363-4372. PubMed
34.Malhotra S, Cheriff AD, Gossey JT, Cole CL, Kaushal R, Ancker JS. Effects of an e-prescribing interface redesign on rates of generic drug prescribing: exploiting default options. J Am Med Inform Assoc. 2016;23(5):891-898. PubMed
35.Colla CH, Mainor AJ, Hargreaves C, Sequist T, Morden N. Interventions aimed at reducing use of low-value health services: a systematic review. Med Care Res Rev. 2017;74(5):507-550. PubMed
36.Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform. 2017;8(3):686-697. PubMed
37.Li E, Manz C, Liu M, et al. Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: secondary analysis of a randomized clinical trial. PLoS One. 2022;17(5):e0267012. PubMed
38.van Walraven C, McAlister FA, Bakal JA, Hawken S, Donzé J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission. CMAJ. 2015;187(10):725-733. PubMed
39.van Walraven C. The Hospital-patient One-year Mortality Risk score accurately predicted long-term death risk in hospitalized patients. J Clin Epidemiol. 2014;67(9):1025-1034. PubMed
40.Wegier P, Koo E, Ansari S, et al. mHOMR: a feasibility study of an automated system for identifying inpatients having an elevated risk of 1-year mortality. BMJ Qual Saf. 2019;28(12):971-979. PubMed
41.HOMR. Empowerment and teamwork at the Kingston Health Sciences Centre. 2023; https://homrinitiative.ca/empowerment-and-teamwork-at-the-kingston-health-sciences-centre/. Accessed 2023 Nov 22.
42.HOMR. For Dr. Martin Chasen at William Osler Health System, HOMR is ‘the future’. 2023; https://homrinitiative.ca/for-dr-martin-chasen-at-william-osler-health-system-homr-is-the-future/. Accessed 2023 Nov 22.
43.van Walraven C, Forster AJ. The HOMR-Now! model accurately predicts 1-year death risk for hospitalized patients on admission. Am J Med. 2017;130(8):991.e999-991.e916. PubMed
44.Chi S, Kim S, Reuter M, et al. Advanced care planning for hospitalized patients following clinician notification of patient mortality by a machine learning algorithm. JAMA Netw Open. 2023;6(4):e238795. PubMed
45.Murphree DH, Wilson PM, Asai SW, et al. Improving the delivery of palliative care through predictive modeling and healthcare informatics. J Am Med Inform Assoc. 2021;28(6):1065-1073. PubMed
46.Wilson PM, Ramar P, Philpot LM, et al. Effect of an artificial intelligence decision support tool on palliative care referral in hospitalized patients: a randomized clinical trial. J Pain Symptom Manage. 2023;66(1):24-32. PubMed
47.Mason J, Morrison A, Visintini S. An overview of clinical applications of artificial intelligence. Can J Health Technol. 2018(174). https://www.cadth.ca/sites/default/files/pdf/eh0070_overview_clinical_applications_of_AI.pdf. Accessed 2023 Sep 26.
48.Walshe C, Todd C, Caress A, Chew-Graham C. Patterns of access to community palliative care services: a literature review. J Pain Symptom Manage. 2009;37(5):884-912. PubMed
49.Rosenwax L, Spilsbury K, McNamara BA, Semmens JB. A retrospective population based cohort study of access to specialist palliative care in the last year of life: who is still missing out a decade on? BMC Palliat Care. 2016;15:46. PubMed
50.Stiel S, Heckel M, Seifert A, Frauendorf T, Hanke RM, Ostgathe C. Comparison of terminally ill cancer- vs. non-cancer patients in specialized palliative home care in Germany - a single service analysis. BMC Palliat Care. 2015;14:34. PubMed
51.Lau C, Meaney C, Morgan M, Cook R, Zimmermann C, Wentlandt K. Disparities in access to palliative care facilities for patients with and without cancer: a retrospective review. Palliat Med. 2021;35(6):1191-1201. PubMed
Note that this appendix has not been copy-edited.
An information specialist conducted a literature search on key resources including MEDLINE, Embase, the Cochrane Database of Systematic Reviews, the International HTA Database, the websites of Canadian and major international health technology agencies, as well as a focused internet search. The search approach was customized to retrieve a limited set of results, balancing comprehensiveness with relevancy. The search strategy comprised both controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings), and keywords. Search concepts were developed based on the elements of the research questions and selection criteria. The main search concepts were artificial intelligence and end of life. The search was completed on September 6, 2023, and limited to English-language documents published since January 1, 2018.
One author screened the literature search results and reviewed the full text of all potentially relevant studies. Studies were considered for inclusion if the intervention was a combination of AI-based mortality predictive models and behavioural interventions targeting clinicians to initiate end-of-life conversations with patients. Conference abstracts and grey literature were included when they provided additional information to that available in the published studies.
Note that this appendix has not been copy-edited.
Table 1: Characteristics and Predicting Performance of the AI Algorithms of the Included Behavioural Nudges
AI-based behavioural nudge | AI-based nudge developed by researchers at University of Pennsylvania4 | |
|---|---|---|
Country | US | US |
Type of disease | All cancer | All cancer |
Algorithm | N-dimensional eigenspace | Gradient-boosted tree |
Input features | Clinical data and billing information (e.g., diagnosis codes, assessments, laboratories, medications, cancer staging information, vitals, and screenings) from the EHR Socioeconomic dataa (e.g., income, household size, transportation) from publicly available resources including US Census Bureau, US Department of Agriculture and the National Oceanic and Atmospheric Administration Behavioural dataa (e.g., history of internet searches on health conditions, purchasing channels and life stage) from third-party data vendors such as Acxiom, Experian, and Transunion | Structured EHR data including demographic, clinicopathologic, laboratory, comorbidity, and electrocardiogram data |
Outcome | 30-day mortality | 180-day mortality |
Validation | Internally validated in 3,671 patients in a community practice in the Pacific Northwest of the US No external validation Threshold for high-risk patients: 5% | Internally validated in 24,582 patients in a tertiary practice within the University of Pennsylvania Health System No external validation Threshold for high-risk patients: 40% |
Area under the receiver operating characteristic curve | 0.86 | 0.89 |
Sensitivity | 0.28 | 0.27 |
Specificity | 0.95 | 0.99 |
Positive predictive value | Not reported | 0.45 |
Negative predictive value | 0.99 | 0.97 |
EHR = electronic health record.
aSocioeconomic and behavioural data were used during model development. It is unclear whether the commercial nudge retrieves patients’ behavioural and socioeconomic data out of the EHR system to predict individual mortality risk in practical use.
Authors: Weiyi Xie, Robyn Butcher
ISSN: 2563-6596
Disclaimer: The information in this document is intended to help Canadian health care decision-makers, health care professionals, health systems leaders, and policy-makers make well-informed decisions and thereby improve the quality of health care services. While patients and others may access this document, the document is made available for informational purposes only and no representations or warranties are made with respect to its fitness for any particular purpose. The information in this document should not be used as a substitute for professional medical advice or as a substitute for the application of clinical judgment in respect of the care of a particular patient or other professional judgment in any decision-making process. The Canadian Agency for Drugs and Technologies in Health (CADTH) does not endorse any information, drugs, therapies, treatments, products, processes, or services.
While care has been taken to ensure that the information prepared by CADTH in this document is accurate, complete, and up-to-date as at the applicable date the material was first published by CADTH, CADTH does not make any guarantees to that effect. CADTH does not guarantee and is not responsible for the quality, currency, propriety, accuracy, or reasonableness of any statements, information, or conclusions contained in any third-party materials used in preparing this document. The views and opinions of third parties published in this document do not necessarily state or reflect those of CADTH.
CADTH is not responsible for any errors, omissions, injury, loss, or damage arising from or relating to the use (or misuse) of any information, statements, or conclusions contained in or implied by the contents of this document or any of the source materials.
This document may contain links to third-party websites. CADTH does not have control over the content of such sites. Use of third-party sites is governed by the third-party website owners’ own terms and conditions set out for such sites. CADTH does not make any guarantee with respect to any information contained on such third-party sites and CADTH is not responsible for any injury, loss, or damage suffered as a result of using such third-party sites. CADTH has no responsibility for the collection, use, and disclosure of personal information by third-party sites.
Subject to the aforementioned limitations, the views expressed herein are those of CADTH and do not necessarily represent the views of Canada’s federal, provincial, or territorial governments or any third-party supplier of information.
This document is prepared and intended for use in the context of the Canadian health care system. The use of this document outside of Canada is done so at the user’s own risk.
This disclaimer and any questions or matters of any nature arising from or relating to the content or use (or misuse) of this document will be governed by and interpreted in accordance with the laws of the Province of Ontario and the laws of Canada applicable therein, and all proceedings shall be subject to the exclusive jurisdiction of the courts of the Province of Ontario, Canada.
The copyright and other intellectual property rights in this document are owned by CADTH and its licensors. These rights are protected by the Canadian Copyright Act and other national and international laws and agreements. Users are permitted to make copies of this document for non-commercial purposes only, provided it is not modified when reproduced and appropriate credit is given to CADTH and its licensors.
About CADTH: CADTH is an independent, not-for-profit organization responsible for providing Canada’s health care decision-makers with objective evidence to help make informed decisions about the optimal use of drugs, medical devices, diagnostics, and procedures in our health care system.
Funding: CADTH receives funding from Canada’s federal, provincial, and territorial governments, with the exception of Quebec.
Questions or requests for information about this report can be directed to Requests@CADTH.ca