RapidAI for Stroke Detection and AI Implementation Review

Key Messages

RapidAI Review for Stroke Detection

What Is the Issue?

• Stroke is a sudden loss of neurologic function caused by poor or interrupted blood flow within the brain. It is 1 of the leading causes of death and a major cause of disability in Canada. For patients with suspected stroke, prompt evaluation using CT imaging and other tests can help to determine the type of stroke, to assess the severity of damage, and to guide treatment decisions.
• RapidAI is an artificial intelligence (AI)–enabled software platform that facilitates the viewing, processing, and analysis of CT images to aid clinicians in assessing patients with suspected stroke. Understanding the potential benefits and harms of using RapidAI is important to clarify its role in stroke detection.

What Did We Do?

• We sought to identify, synthesize, and critically appraise literature evaluating the effectiveness, accuracy, and cost-effectiveness of RapidAI for detecting large-vessel occlusion (LVO) (i.e., ischemic stroke) and intracranial hemorrhage (ICH) (i.e., hemorrhagic stroke).
• We searched key resources, including journal citation databases, and conducted a focused internet search for relevant evidence published up to July 22, 2024. We screened citations for inclusion based on predefined criteria, critically appraised the included studies, narratively summarized the findings, and assessed the certainty of evidence. Our methods were guided by the Scottish Health Technologies Group’s health technology assessment (HTA) framework.
• We highlighted and reflected on the ethical and equity implications of using RapidAI for stroke detection, found in the clinical literature, integrating these considerations throughout the review.
• We engaged a patient contributor who had experienced a hemorrhagic stroke, to learn about her experience, perspectives, and priorities. Additionally, we incorporated feedback from clinical and ethics experts, the manufacturer, and other interested parties.

What Did We Find?

• We found 2 cohort studies and 11 diagnostic accuracy studies that assessed the effectiveness and accuracy of RapidAI for detecting stroke. Among these, 3 studies evaluated RapidAI as it is intended to be used in clinical practice (i.e., to complement clinician interpretation of CT images), while the remaining 10 studies assessed RapidAI as a standalone intervention.
• The patient contributor identified important outcomes for stroke care, including improving speed and accuracy of diagnosis, minimizing the damaging effects of stroke, and reducing mortality rates. She also highlighted ethical considerations regarding the use of AI in health care, such as providing data privacy and equitable access, as well as informing patients about the use of AI technologies in the care pathway.
• Low-certainty evidence suggests that evaluation of CT angiography images by Rapid LVO combined with clinician interpretation, compared to clinician interpretation alone, may result in clinically important reductions in radiology-report turnaround time in patients with suspected stroke. For detecting ICH, low-certainty evidence suggests that Rapid ICH combined with clinician interpretation, using clinician interpretation as a reference standard, has a sensitivity of 92% (95% confidence interval [CI], 78% to 98%) and a specificity of 100% (95% CI, 98% to 100%). However, estimates of sensitivity and specificity for detecting LVO varied, based on studies using different modules of RapidAI as a standalone intervention, providing only indirect accuracy data.
• The effects of RapidAI on other time-to-intervention metrics, measures of physical and cognitive function, and response to therapy (e.g., reperfusion rates) were very uncertain. We did not identify any evidence on the effects of RapidAI on many important clinical outcomes, including patient harms, mortality, health-related quality of life, length of hospital stay, or health care resource implications.
• We did not find any studies on the cost-effectiveness of RapidAI for detecting stroke that met our selection criteria for this review.
• Ethical and equity considerations related to patient autonomy, privacy, transparency, access, and algorithmic bias have implications across the technology life cycle when using RapidAI for detecting stroke.

What Does This Mean?

• RapidAI has the potential to improve acute stroke care by creating efficiencies in the diagnostic process. However, the impact of RapidAI on many outcomes, including those that are important to patients, is uncertain due to limitations of the available evidence.
• To improve the certainty of findings, there is a need for evidence from robustly conducted studies at lower risk of bias that enrol diverse patient populations and measure outcomes that are important to patients, with improved reporting.
• The cost-effectiveness of RapidAI for stroke detection is currently unknown.
• In addition to the evidence on the effectiveness and accuracy of RapidAI for detecting stroke, decision-makers may wish to reflect on the ethical and equity considerations that arise during the deployment of AI-enabled technologies, such as those related to autonomy, privacy, transparency, and explainability of machine-learning models, and the need for considerations related to equity and access in their design, development, and deployment.

AI Implementation Review

What Is the Issue?

• Globally, we are seeing a widespread increase in the interest, development, and use of artificial intelligence (AI)–enabled medical devices. Comprehensive evaluation through health technology assessment (HTA) can ensure that digital health technologies (DHTs), including AI-enabled medical devices, are adequately equipped to balance benefits and harms, while being interoperable and equitably accessible to people living in Canada.
• In the UK, a checklist called Digital Technology Assessment Criteria (DTAC) is used as an add-on component to HTAs to capture additional considerations for the implementation of DHTs. The 5 core areas of DTAC are clinical safety, data protection, technical security, interoperability, and usability and accessibility. In Canada, we currently do not have a DTAC equivalent that can be used as an add-on to traditional HTA.
• This implementation review is needed to assist health systems in Canada in preparing for the uptake of AI-enabled medical devices, as these technologies pose new challenges. We assessed whether the safeguards and assessment criteria captured by DTAC and other AI-related resources are in place to inform decision-making around the digital infrastructure elements of implementation.

What Did We Do?

• We conducted an implementation review, using a phased approach, to determine whether DTAC can be applied to the health care context in Canada to inform the implementation of DHTs and to identify any additional implementation considerations specific to the use of AI-enabled medical devices in Canada. We integrated ethics and equity considerations across both phases of the review.
• In phase 1, we applied DTAC to the health care context in Canada by determining whether we have equivalent or similar measures, strategies, and policies in place to implement DHTs safely.
• In phase 2, an information specialist searched for literature to identify implementation guidance specific to AI and relevant to Canada to supplement DTAC. One reviewer screened publications for inclusion based on predefined criteria, incorporated relevant information into tables, and summarized the findings narratively.
• We leveraged patient engagement activities conducted in a concurrent Canada’s Drug Agency review of a specific AI-enabled medical device in stroke detection to learn from a patient contributor with lived experience of a hemorrhagic stroke. We learned about her experience, perspectives, priorities, and thoughts about using AI in clinical decision-making.

What Did We Find?

• With some caveats, we found that many of DTAC’s assessment criteria have equivalent or similar guidance for the health care context in Canada. Some exceptions are derived from the differences in Canada’s current governance and health care structure. Further investigation is required to understand whether certain policies in Canada provide sufficient coverage to fulfill DTAC’s criteria (e.g., clinical safety).
• We identified several considerations for implementing AI-enabled medical devices, with many having underlying ethical and equity implications. Much of the identified guidance emphasizes implementation considerations that apply to the AI system’s entire life cycle, including the most prevalent consideration: ensuring AI-enabled medical devices are monitored, maintained, and sustainable. Examples of additional considerations include AI data governance and data protection; transparency and explainability; and inclusiveness, equity, and minimization of bias.
• The patient contributor highlighted several considerations relevant for this review, such as data protection and privacy as well as accessibility and equity.

What Does This Mean?

• We have identified key considerations for AI-enabled medical devices that health care decision-makers may consider for the safe and successful implementation of AI in health care in Canada.
• While Canada has DTAC-equivalent or similar measures, strategies, or policies in place, we identified a need for a checklist like DTAC that senior decision-makers can use. This checklist could be an adaptation of DTAC and could include additional implementation considerations for AI-enabled medical devices to ensure that these technologies meet the minimum baseline standards set out by DTAC and inform the next steps for the safe and successful implementation of AI-enabled medical devices in Canada.
• This implementation review for all AI-enabled medical devices is to be used alongside reviews of specific AI technologies, including the concurrent review of RapidAI, and will serve as a foundational report to be tailored for each AI topic and updated with the latest developments in the regulation and other aspects of management of AI in the context of Canada.