CADTH Health Technology Review

Drug Shortages: A Scoring System to Evaluate and Prioritize the Clinical Importance of Drugs in Canada

Environmental Scan

Key Messages

• Drug shortages are an ongoing global health issue and pose significant challenges to policy-makers and health care providers. They may result from various factors (e.g., supply chain disruptions and recalls) and have a negative impact on patient outcomes and health care expenditures.

• The ability to accurately predict drugs at risk of a shortage would be a vital aspect in the effective management of drug shortages and help ensure proactive measures for consistent drug supply and optimized patient care; however, these drugs must also be prioritized in a meaningful way to identify high priority targets among the large number of drugs in the market and help ensure resources are allocated to the most important at-risk drugs.

• The purpose of this Environmental Scan aimed to identify a systematic method of objectively evaluating and ranking the value of drugs based on important clinical criteria and adapt the selected tool to prioritize the clinical importance of drugs in Canada.

• A total of 5 potential scoring systems were considered, and 1 scoring system was selected and adapted (based on expert panel member feedback). Finally, an assessment of 9 clinical criteria was made across 5 broad categories. The criteria include an evaluation of disease severity, size of the population, clinical guidelines, comparator interventions limitations (unmet needs), improvement of efficacy or effectiveness, improvement of safety and tolerability, improvement of patient-reported outcomes (PRO), type of medical service (e.g., symptom relief, prolonging life, cure), and public health interest.

• The proposed tool can be used in situations where decision-making requires an objective and systematic evaluation of drugs' clinical importance. Additionally, it will help inform a drug shortage risk framework in Canada, which will consider both the clinical importance and supply chain risks of drugs as part of the Health Canada Drug Shortages Task Force.

Context

Drug shortages are an ongoing global health issue and pose significant challenges to policy-makers and health care providers.^1-6 They may result from various factors, including supply chain disruptions, unexpected increases in demand, and globalization, and can have a negative impact on patient outcomes.^1-7 Drug shortages can also have significant financial implications, both direct and indirect, primarily driven by increased drug acquisition and personnel expenditures associated with identifying and acquiring therapeutic alternatives.^4-6,8 Though certain types of drugs are more vulnerable to shortages, they have occurred across a range of different medications, and any given drug has the potential to be at risk.^4,5,7,9,6,7Between 2022 and 2023 in Canada, more than 2,700 drug shortages were reported to Health Canada (HC), which lasted an average of 98 days.⁷ As such, HC has created a Health Products Shortages Directorate to identify how to address health product shortages better.⁶

Though there are currently measures in place to prevent, mitigate, and resolve drug shortages,^1,2,5-7 the ability to predict drugs at risk of a shortage accurately can be another helpful strategy used to reduce their occurrence and duration, as lack of advanced warning has been identified as a leading cause of this problem.^4,9-11 This would reduce the need to depend on manufacturers or distributors to report an impending drug shortage and prevent situations where activities to manage a drug shortage only occur when it begins.^4,5,9,10 The ability to predict which drugs have the highest probability of shortage in advance will help ensure proactive measures for consistent drug supply and optimized patient care.^9,11

In addition to accurately predicting drugs at risk of a shortage, incorporating a systematic approach to prioritizing these drugs based on clinical importance would be helpful in effectively managing drug shortages, as this would help identify high-priority targets among the abundance of drugs in the market.^12,13 Quantitative methods to objectively measure the value of drugs in decision-making have previously been investigated across a range of therapeutic areas using various frameworks, which typically comprise multiple assessment components including, but not limited to, important clinical criteria.^14-18 Such tools could be leveraged to manage drug shortages to help allocate resources to the most clinically important at-risk drugs.

This Environmental Scan (ES) is part of a series of initiatives regarding drug shortages and was conducted to identify and adapt a scoring system that could be used to rank the importance of drugs systematically and objectively in Canada based on clinical criteria. Additionally, this ES was completed to support an expert panel involved in developing a drug shortage risk framework, which considers both the drugs' clinical importance and supply chain risks.^6,19

Objectives

The key objectives of this ES were as follows:

1. To identify scoring systems developed to evaluate the importance of drugs based on clinical criteria.

2. To identify a single scoring system, adapt it to rank the clinical importance of drugs, and inform a drug shortage risk prediction model in Canada.

Research Questions

1. Are there scoring systems that exist that were developed to evaluate the importance of drugs based on clinical criteria?

2. Is there a scoring system that can be used and adapted to rank the clinical importance of drugs and inform a drug shortage risk prediction model in Canada?

Methods

Literature Search

An information specialist conducted a literature search on key resources, including MEDLINE, the websites of Canadian and major international health technology agencies, and a focused internet search. The search strategy comprised controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings) and keywords (Appendix 1). The search was completed on September 15, 2023, and limited to English-language documents published since 2003.

Screening and Study Selection

One reviewer screened and selected studies for inclusion. In the first screening level, titles and abstracts were reviewed, and potentially relevant articles were retrieved and assessed for inclusion. The final selection of full-text articles was based on the criteria presented in Table 1. Screening focused on studies which utilized specific scoring systems and well-defined clinical criteria for effective drug prioritization based on clinical characteristics associated with clinical importance that were not specific to an indication or drug class. Information was extracted on study characteristics and methodology, the clinical criteria used in the scoring system, and how aggregate scores were calculated. No formal critical appraisal of the included studies was conducted.

Table 1: Article Inclusion Criteria

Selection and Adaptation of Single Scoring System for Drug Shortage Risk Framework

This report provides an overview of studies that employed specific scoring systems, focusing particularly on clinical criteria to rank drugs' importance effectively. After identifying potential scoring systems, they were compared qualitatively, and a single scoring system was selected to inform the drug shortage risk framework based on a comprehensive assessment of the clinical criteria included in the scoring system and scoring methodology.

After a single scoring system was selected, it was piloted and adapted with the involvement of an expert panel and clinicians as part of the clinical review process. Expert panel members assessed the scoring system categories and definitions, providing suggestions based on individual expertise. The tool was then tested by 3 clinicians, with changes adapted to ensure interpretability and ease of use. After revising the tool, clinicians with diverse expertise and experience used the selected scoring system to evaluate a list of drugs. Additional adjustments were made to refine the final scoring system within the clinical review process based on issues that arose in specific cases. The final scoring system was built from the ES and adapted based on expert panel and clinician review to improve the use of the system as part of the drug shortages risk framework. Further details of the use of the tool and integration into the framework will be described in future reports by the team leading the development of the framework.

Findings

Literature Search Results

A total of 447 citations were identified from the MEDLINE search. Following title and abstract screening, 96 potentially relevant reports were retrieved for full-text review. Of these potentially relevant articles, 3 were included in this report.^14,20,21 Additionally, 2 other articles identified from the focused internet search were also included,^22,23 resulting in a total of 5 included articles.

Summary of Identified Scoring Systems

An overview of the identified scoring systems, including their objectives, criteria selection methods, and selected criteria, is presented in Appendix 2. Overall, the identified scoring systems collectively underscored the significance of adopting the multicriteria decision analysis (MCDA) framework as a pivotal methodological approach within health technology assessment (HTA) and drug evaluation practices. The scoring systems used various methodologies to identify and weigh criteria for the MCDA models. Angelis and Kanavos highlighted stages, including a systematic literature review (SLR), expert consultation, and value tree development.²² Goetghebeur et al. piloted an adaptable decision-making framework incorporating MCDA in HTA.²⁰ Iskrov et al. involved a survey to identify decision criteria and stakeholder preferences in the HTA of orphan drugs, developing a simple MCDA model reflecting societal perspectives through stakeholder input.²¹ Jakab et al. conducted an SLR to identify and reduce criteria for future MCDA frameworks.¹⁴ Zelei et al.’s literature review emphasized novel criteria in pricing and reimbursement decisions for pharmaceuticals.²³

The selection criteria used across the identified scoring systems are presented in Table 2. Across the scoring systems, broad overarching categories were identified, which were used to organize the revised MCDA criteria framework: Disease Impact, Efficacy, Safety and Tolerability, Contextual Considerations including existing guidelines and availability of comparator interventions, criteria defining additional Patient Benefit and Other Benefits (i.e., societal, health-systems-based, innovation, and other considerations), and quality of evidence.

Table 2: Selection Criteria Used in the Identified Scoring Systems

Among the identified selection criteria, the following themes emerged:

• Disease Impact: All 5 scoring systems emphasized disease severity, its impact on a patient population, and the burden of the disease. Differences between scoring systems included Angelis and Kanavos also emphasizing unmet need,²² while Iskrov et al. distinguishes between chronic life-threatening and nonlife-threatening disorders.²¹

• Clinical Effectiveness, Patient and Other Benefits: The evaluation of therapeutic impact, improvement in efficacy or effectiveness, and health benefits, including life expectancy and quality of life enhancement, are consistent across studies. Factors such as patient adherence, persistence, ease of use, and added value services to patients also underscored the importance of patient-centred outcomes and experiences. While the scoring systems uniformly assess efficacy end points, Zelei et al. and Iskrov et al. specifically distinguish interventions based on lifesaving attributes.^21,23 Moreover, the scoring systems encompass diverse patient benefits, illustrating wide-ranging impact assessments.

• Safety and Tolerability: Criteria related to safety profiles and adverse effects highlighted the criticality of assessing interventions' safety, tolerability, and efficacy. Iskrov et al. specifically distinguish between adverse effect frequency.²¹ Safety and Tolerability considerations as a separate criterion were not explicitly detailed or discussed in the scoring system by Jakab et al.¹⁴

• Contextual Considerations: Understanding the context of interventions about existing guidelines and the availability of alternative therapies is a recurring theme for many of these scoring systems, reflecting the importance of considering the broader treatment landscape; however, not all studies included contextual considerations such as alignment with existing guidelines, expert opinion, or availability of comparator or alternative treatment options. Iskrov et al. and Jakab et al. focused primarily on alternative treatment availability,^14,21 while Goetghebeur et al. and Zelei et al. incorporated both contextual considerations (i.e., both expert opinion or guidelines and comparator interventions or alternative treatments).^20,23

• Quality of Evidence: Assessment of the quality of evidence also holds significance in drug evaluation and decision-making. Quality of Evidence considerations as a separate criterion were not explicitly detailed or discussed in the scoring system by Angelis and Kanavos.²²

Selection and Adaptation of a Single Scoring System

The 5 identified scoring systems underwent further assessment to identify those that presented a comprehensive approach, covering all proposed assessment criteria of interest without any notable gaps (Table 2). Specifically, this evaluation focused on determining the extent to which each study thoroughly addressed the categories of interest (i.e., Disease Impact, Intervention Outcomes: Efficacy, Safety and Tolerability, Patient Benefit, and so forth.) within the context of the MCDA. Among the 5 scoring systems, 2 studies, Goetghebeur et al. and Zelei et al.,^20,23 were deemed to have a thorough assessment, covering all categories within the MCDA matrix, and were considered for selection and adaptation for informing the drug shortage risk framework.

Subsequently, it was examined whether the weighting or scoring of each criterion was further investigated or proposed in the 2 scoring systems under consideration. Indeed, both Goetghebeur et al. and Zelei et al. described the weighting of their proposed assessment criteria. Table 3 presents the weight or relative importance of the top 5 criteria in each scoring system. For additional details on the “Weight or Relative Importance of Clinical Criteria,” refer to Appendix 2. Regarding the proposed weighting score, Zelei et al. conducted an SLR focusing on MCDA and its use in evaluating therapies specifically for rare diseases.²³ In contrast, the method suggested by Goetghebeur et al. comprised a 2-part evaluation process based on the Evidence and Value: Impact on Decision-Making (EVIDEM) framework, specifically developed to bridge HTA with MCDA. This scoring system was designed to provide a core MCDA model adaptable to the context of decision-makers using the tool, using a contextual tool combined with a “by-criterion” HTA report methodology to provide synthesized evidence at the criterion level; these multipurpose tools consist of an MCDA module and an HTA module.²⁰ Thus, the scoring system by Goetghebeur et al. was considered a cornerstone study within its domain, given its comprehensive approach and preliminary weighting assessment validation, and was selected to be used and adapted to objectively rank the clinical importance of drugs and inform the drug shortage risk prediction model.

Table 3: Weight or Relative Importance of the Top 5 Criteria in the Scoring Systems of Goetghebeur et al. and Zelei et al.

For each scoring system, the top 5 criteria are indicated and ranked from 1 to 5, with 1 representing the criterion with the highest weight or importance.

^aTie in the ranking of 2 criteria.

As outlined in the study by Goetghebeur et al. (Appendix 3),²⁰ decision criteria on which the framework was built were identified based on an extensive analysis of the literature and detailed analysis of drug coverage decision-making processes from more than 20 jurisdictions worldwide. The MCDA model was subsequently built by selecting a set of criteria that fulfilled the principles of MCDA modelling (i.e., completeness, nonredundancy, operationality, and mutual independence), consisting of criteria (quantifiable or intrinsic criteria) that are universally operationalizable (i.e., low and high ends of scales are widely agreed upon). Preliminary validation and testing of this decision framework were also performed. A group of representative health care stakeholders appraised the medicines as case studies from several therapeutic areas by creating detailed HTA reports. These reports, drawn from extensive literature and proprietary data on prior submissions, informed the decision criteria in an MCDA matrix. Stakeholders first weighted the importance of each criterion of the MCDA core model independent of the interventions to be appraised. They were asked to independently weigh the importance of each criterion to reflect their perspective in the context of a decision-making committee to optimize health at the societal level. For this purpose, a simple weight elicitation technique on a 5-point scale was used, with 1 representing the most minor important criteria. They then scored the medicines accordingly within the matrix, collecting feedback through structured discussions. To appraise a drug, each criterion is scored on a 4-point scale (minimum score = 0 and maximum score = 3). Relative weights varied widely among participants, although there appeared to be good consensus on the importance of “improvement of efficacy or effectiveness” and “relevance and validity of evidence.” The top 5 criteria allocated the highest weights were, in order: “relevance and validity of evidence,” “improvement of efficacy or effectiveness,” “public health interest,” “improvement of safety and tolerability,” “type of medical service (cure)” and “completeness and consistency of reporting evidence” (there was a tie between the last 2 criteria).

The scoring system by Goetghebeur et al. was then adapted using feedback obtained by expert panel members and clinicians as part of the development of a risk framework and at-risk medicines list. The revised decision matrix is presented in Table 4. This revised matrix emphasizes 9 clinical criteria of interest across 5 categories. The proposed matrix underwent adaptation to capture distinct categories more effectively within MCDA. This adaptation involved integrating grouped criteria proposed from prior studies and addressing areas of overlap among identified MCDA criteria found in the literature. Of note, this suggested framework serves as an initial guide only and may be further refined through additional assessments and adaptations in developing an algorithm for drug shortages. Specific adaptations included:

• Attaching a conflict of interest note for consideration within the clinician review process under C1 (“Clinical guidelines”).

• Specifying the availability and convenience of medications within Canada under C2 (“Comparator intervention limitations”).

• Building on the details surrounding the condition's association with the timing and occurrence of severe outcomes under D1 (“Disease Severity”).

• Removal of criteria related to the categories of “economics” and “quality of evidence” (criteria E1-E3 and Q1-Q3 in the model proposed by Goetghebeur et al.), as the focus of this work was on drug evaluation based on clinical criteria only.

• Modifying the description of the minimum score for criteria I1 and I2.

• Categorizing criterion P1 (“Improvement of patient-reported outcomes”) under “Patient Benefit” (Goetghebeur et al. categorized this criterion under “Intervention Outcomes,” I3) and modifying the description of its minimum score.

• Categorizing criterion P2 (“Type of medical service (cure)”) under “Patient Benefit” (Goetghebeur et al. categorized this criterion under “Type of benefit,” T2).

• Changing the minimum score to −1 for criteria C2 (“comparator interventions limitations (unmet needs)”) and P2 (“type of medical service (cure)”). These changes were made to consider interventions that result in worse patient outcomes or offer no benefit or improvement.

• Adding considerations for the impact on marginalized populations’ inequality if the drug was unavailable under T1 (“public health interest).

Table 4: Proposed Scoring System to Rank the Clinical Importance of Drugs in Canada (Adapted from Goetghebeur et al.)

COI = conflict of interest.

Limitations

A limitation of this report is that this was not a comprehensive SLR. Only 1 electronic database was searched, and the search was restricted to publications within the past 20 years of the search date; therefore, other potentially relevant articles outside these search parameters were missed. Additionally, the objective of the selected scoring system by Goetghebeur et al. was to develop an MCDA framework for the purposes of decision-making within the context of HTA and not specifically for informing the management of drug shortages; however, Goetghebeur et al.’s intention was that their framework could be adapted to the context of the decision-makers using the tool, and expert panel members and clinicians piloted the tool to ensure its applicability to prioritizing the clinical importance of drugs in Canada and modified according to their feedback. Lastly, no formal critical appraisal of the methodologies of the included scoring systems was completed.

Conclusions and Implications for Decision or Policy-Making

The ability to accurately predict drugs at risk of a shortage would be a vital aspect in effectively managing drug shortages; however, these drugs must also be prioritized in a meaningful way to identify high-priority targets among the abundance of drugs in the market. The purpose of this ES was to identify a systematic method of objectively evaluating and ranking the value of drugs based on critical clinical criteria and adapt the selected tool to the context of prioritizing the clinical importance of drugs in Canada. A total of 5 potential scoring systems were considered and 1 scoring system was selected and adapted (based on expert panel member and clinician feedback), finally comprising an assessment of 9 clinical criteria across 5 different categories. The criteria include an evaluation of disease severity, size of the population, clinical guidelines, comparator interventions limitations (unmet needs), improvement of efficacy or effectiveness, improvement of safety and tolerability, improvement of PRO, type of medical service (e.g., symptom relief, prolonging life, cure), and public health interest. The proposed tool can be used when decision-making requires an objective and systematic evaluation of the clinical importance of drugs. Additionally, this tool will help inform a drug shortage risk framework and at-risk medicines list in Canada, which will consider both the clinical importance and supply chain risks of drugs.

References

1.Protocol for the Notification and Communication of Drug Shortages. Drug Shortages Canada; 2017: https://www.drugshortagescanada.ca/files/MSSC_Protocol_2017.pdf. Accessed April 14, 2024.

2.A Toolkit for Improved Understanding and Transparency of Drug Shortage Response in Canada. Drug Shortages Canada; 2017: https://www.drugshortagescanada.ca/files/MSSC_Toolkit_2017.pdf. Accessed April 14, 2024.

3.Aronson JK, Heneghan C, Ferner RE. Drug shortages. Part 1. Definitions and harms. Br J Clin Pharmacol. 2023;89(10):2950-2956. PubMed

4.Ventola CL. The drug shortage crisis in the United States: causes, impact, and management strategies. P t. 2011;36(11):740-757. PubMed

5.Shukar S, Zahoor F, Hayat K, et al. Drug Shortage: Causes, Impact, and Mitigation Strategies. Front Pharmacol. 2021;12:693426. PubMed

6.Health Canada. What we heard: Improving access to drugs and other health products in Canada. Government of Canada. https://www.canada.ca/en/health-canada/services/drugs-health-products/drug-products/drug-shortages/what-we-heard-report.html. Published 2023. Accessed April 20, 2024.

7.Drug Shortages in Canada. Ottawa (ON): Health Canada; 2023: https://www.canada.ca/content/dam/hc-sc/documents/services/drugs-health-products/drug-products/drug-shortages/2022-2023-review/2022-2023-review-en.pdf. Accessed April 14, 2024.

8.Fox ER, Birt A, James KB, Kokko H, Salverson S, Soflin DL. ASHP Guidelines on Managing Drug Product Shortages in Hospitals and Health Systems. Am J Health Syst Pharm. 2009;66(15):1399-1406. PubMed

9.Liu I, Colmenares E, Tak C, et al. Development and validation of a predictive model to predict and manage drug shortages. Am J Health Syst Pharm. 2021;78(14):1309-1316. PubMed

10.Jenks S. Efforts Underway To Curb Drug Shortages. JNCI: Journal of the National Cancer Institute. 2011;103(12):914-915. PubMed

11.Pall R, Gauthier Y, Auer S, Mowaswes W. Predicting drug shortages using pharmacy data and machine learning. Health Care Manag Sci. 2023;26(3):395-411. PubMed

12.Socal MP, Estus E, Long J, Crane MA, Pegany V, Anderson GF. Developing Prioritization Criteria to Identify Target Drugs for CalRx, the California Generic Drugs Initiative. Value in Health. 2023;26(5):634-638. PubMed

13.Esba LCA, Almodaimegh H, Alhammad A, Ferwana M, Yousef C, Ismail S. P&T Committee Drug Prioritization Criteria: A Tool Developed by a Saudi Health Care System. P t. 2018;43(5):293-300. PubMed

14.Jakab I, Németh B, Elezbawy B, et al. Potential Criteria for Frameworks to Support the Evaluation of Innovative Medicines in Upper Middle-Income Countries-A Systematic Literature Review on Value Frameworks and Multi-Criteria Decision Analyses. Front Pharmacol. 2020;11:1203. PubMed

15.Bentley TGK, Cohen JT, Elkin EB, et al. Measuring the Value of New Drugs: Validity and Reliability of 4 Value Assessment Frameworks in the Oncology Setting. J Manag Care Spec Pharm. 2017;23(6-a Suppl):S34-s48.

16.Glaus CEG, Kloeti A, Vokinger KN. Defining 'therapeutic value' of medicines: a scoping review. BMJ Open. 2023;13(12):e078134. PubMed

17.Pignatti F, Wilking U, Postmus D, Wilking N, Delgado J, Bergh J. The value of anticancer drugs — a regulatory view. Nature Reviews Clinical Oncology. 2022;19(3):207-215. PubMed

18.Zhang C, Ma EL, Liu BL, Wu B, Gu ZC, Lin HW. Framework Development for Clinical Comprehensive Evaluation of Drugs-a Study Protocol Using the Delphi Method and Analytic Hierarchy Process. Front Pharmacol. 2022;13:869319. PubMed

19.Canada's Drug and Health Technology Agency (CADTH). Drug Shortages in Canada — Expert Panel. https://www.cadth.ca/drug-shortages-canada-expert-panel. Published 2023. Updated October 5, 2023. Accessed April 14, 2024.

20.Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Making. 2012;32(2):376-388. PubMed

21.Iskrov G, Miteva-Katrandzhieva T, Stefanov R. Multi-Criteria Decision Analysis for Assessment and Appraisal of Orphan Drugs. Front Public Health. 2016;4:214. PubMed

22.Angelis A, Kanavos P. Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: The Advance Value Framework. Soc Sci Med. 2017;188:137-156. PubMed

23.Zelei T, Mendola ND, Elezbawy B, Németh B, Campbell JD. Criteria and Scoring Functions Used in Multi-criteria Decision Analysis and Value Frameworks for the Assessment of Rare Disease Therapies: A Systematic Literature Review. Pharmacoecon Open. 2021;5(4):605-612. PubMed

Appendix 1: Search Strategy

Note that this appendix has not been copy-edited.

Database(s): Ovid MEDLINE(R)

Table 5: Literature Search Strategy

Appendix 2: Summary of Identified Scoring Systems

Note that this appendix has not been copy-edited.

Summary of Objectives

1. Angelis and Kanavos, 2017²²

• This paper proposes using MCDA as a methodological approach for evaluating new medicines in HTA and beyond. The focus is on building a multicriteria model to aid in decision-making processes. The paper addresses the need for robust “value frameworks” in assessing new medicines, especially given the escalating drug prices. It notes that existing frameworks often rely on weak methodologies, potentially leading to misleading decisions.

2. Goetghebeur et al., 2012²⁰

• The study aimed to pilot an adaptable decision-making framework incorporating MCDA in HTA. It involved a pan-Canadian group of policy and clinical decision-makers and researchers appraising 10 medicines across 6 therapeutic areas.

3. Iskrov et al., 2016²¹

• This study discusses the use of MCDA in the context of HTA, particularly for orphan drugs. The study addresses the challenges in priority setting and resource allocation, particularly in orphan drugs, where access to market-approved therapies remains an issue. The paper aimed to develop an MCDA value measurement model to assess and appraise orphan drugs by exploring stakeholders' preferences on decision criteria’s weights and performance scores.

4. Jakab et al., 2020¹⁴

• The paper provides a systematic review of criteria used in MCDA for health care, particularly in the context of upper-middle-income countries. The review aimed to develop future MCDA frameworks by proposing criteria focusing on the purchasing decisions of single-source innovative pharmaceuticals in upper-middle-income countries.

5. Zelei et al., 2021²³

• The paper looks at MCDA and its use in evaluating therapies for rare diseases, focusing on novel value criteria typically not captured in traditional cost-effectiveness and budget impact analyses. Objective to perform an SLR of criteria and scoring functions used in value frameworks and MCDA tools for evaluating rare disease therapies.

Summary of Criteria Selection Methods

1. Angelis and Kanavos, 2017²²

• SLR and Expert Consultation (stages 1 and 2): The initial stages involved an SLR and expert consultation in HTA. These steps were crucial for identifying the primary value dimensions considered in the evaluation processes of HTA bodies across different countries.

• Incorporation of Additional Findings (stage 3): The research team incorporated findings from other relevant literature, supplementing the initial results from the SLR and expert consultation.

• Development of the Value Tree Structure (stage 4): The fourth stage focused on developing the structure of the value tree, which would encompass the identified value dimensions and criteria. This stage involved further consultation and validation to ensure the tree's comprehensiveness and practical usefulness.

• Refinement of Criteria and Attributes (stage 5): The final stage involved refining and validating the criteria and attributes that formed the bottom level of the value tree. This stage was essential to ensuring that the value tree was well-structured and captured all essential value concerns of decision-makers in HTA.

2. Goetghebeur et al., 2012²⁰

• The EVIDEM framework, consisting of an MCDA and HTA modules, was used. This framework was developed based on extensive literature analysis and a review of drug coverage decision-making processes in more than 20 jurisdictions globally.

• A diverse panel was convened, including decision-makers, specialists, general practitioners, nurses, pharmacists, and health economists/epidemiologists. This group was tasked with weighting each MCDA decision criterion and scoring medicines concerning each criterion based on synthesized data organized into the MCDA matrix.

3. Iskrov et al., 2016²¹

• A simple MCDA linear additive model was chosen for its value to health authorities and payers. The model combined weighted performance scores of all relevant criteria, reflecting a societal perspective.

• 15 decision criteria were identified through a previous survey and categorized into 3 groups: health technology characteristics, indicated disorder characteristics, and public health aspects. Most criteria were qualitative.

• An online survey collected stakeholders' preferences on each criterion's importance and performance scores. A 100-point scale was used to elicit both weight and performance scores.

• The preliminary model was piloted during a focus group discussion, where stakeholders debated and reviewed the criteria, weights, and scores.

4. Jakab et al., 2020¹⁴

• From 1,878 articles screened, 36 were included in the final data extraction phase, from which 394 criteria were initially extracted. After deduplication and clustering, 26 different criteria were identified. These criteria were then further merged and reduced, resulting in a final set of 16 general criteria proposed for use in the MCDA framework.

5. Zelei et al., 2021²³

• An SLR covered scientific and grey literature on value frameworks and MCDA tools, focusing on pricing and reimbursement decisions for pharmaceuticals. This review aimed to identify and describe applied criteria and scoring functions, particularly emphasizing novel value criteria​. Included studies that used explicit scoring functions for their evaluation criteria and were either orphan drug-specific or considered “referenced” general frameworks.

Summary of Selected Criteria

1. Angelis and Kanavos, 2017²²

• Burden of Disease

  • Unmet need and disease severity

• Therapeutic Impact

  • Direct/Meaningful End points

  • Indirect/Surrogate end points

• Safety Profile

  • Safety and tolerability

  • Contraindications and warnings

• Innovation

  • Spill-over effect^a

  • Mechanism of action

  • Patient convenience

^a Defined as “R&D positive externalities that can lead to the development of subsequent innovation(s).”

2. Goetghebeur et al., 2012²⁰

• Disease Impact

  • Disease severity

  • Size of population

• Context of intervention

  • Clinical guidelines

  • Comparator intervention limitations (unmet needs)

• Intervention outcomes

  • Improvement of efficacy/effectiveness

  • Improvement of safety and tolerability

  • Improvement of PRO

• Type of benefit

  • Public health interest

  • Type of medical service (cure)

• Quality of Evidence

  • Adherence to requirements of the decision-making body

  • Completeness and consistency of reporting evidence

  • Relevance and Validity of evidence

3. Iskrov et al., 2016²¹

• Health benefits

  • Cure

  • Life expectancy and quality of life

  • Life expectancy

  • Quality of life

• Clinical effectiveness

  • Clinical and statistical significance

  • Clinical significance

  • Statistical significance

• Lifesaving

  • Yes

  • No

• Safety

  • Rare adverse effects

  • Nonrare adverse effects

  • Frequent adverse effects

  • Very frequent adverse effects

• Alternative

  • No

  • Yes

• Disease Severity

  • Chronic, life-threatening disorder

  • Acute disorder

  • Chronic nonlife-threatening disorder

• Strength of evidence

  • Randomized controlled clinical trials

  • Nonrandomized clinical trials

  • Cohort and case-control studies

  • Cross-sectional studies

  • Case reports and expert opinions

• Vulnerable groups

  • Indication in children and older people

  • Indication in children

  • Indication in older people

4. Jakab et al., 2020¹⁴

• Disease-related

  • Severity of disease

  • Availability and accessibility of alternative therapies

  • Public health priority

• Treatment-related

  • Health gain

  • Strength of evidence

• Societal

  • Equity

  • Productivity

• Uniqueness and complexity

  • Innovative profile of treatment

  • Manufacturing complexity

• Patient experience

  • Patient adherence and persistence

  • Value-added services to patients

5. Zelei et al., 2021²³

• ISPOR value flower

  • Core criteria

  • Common but inconsistently included criteria

  • Novel criteria

• Incremental cost-effectiveness ratio value framework

• Second Panel on Cost-Effectiveness impact inventory

• Relevant criteria under consideration included:

  • Comparative clinical effectiveness

  • Comparative safety/adverse events/side effects

  • Health-related quality of life and PRO effects

  • Longevity effects

  • Adherence-improving factors (posology, patient convenience, comfort)

  • Unmet medical need/availability of alternative treatments

  • Severity of disease

  • Reduction in uncertainty (strength/quality/level of evidence)

  • Type of therapeutic benefit (curative, lifesaving)

  • Size of population/prevalence

  • Burden of illness

  • Level of innovation

  • Equity/support vulnerable groups

  • Manufacturing complexity

  • Expert opinion

  • Production of toxic waste pollution by intervention

Summary of Weight or Relative Importance of Clinical Criteria

1. Angelis and Kanavos, 2017²²

   • Not applicable: theoretical applications of weigh-score-based criteria discussed, especially in the context of prior research.

2. Goetghebeur et al., 2012²⁰

   • Relative weights varied widely among participants, although there appeared to be good consensus on the importance of:

     • “I1, improvement of efficacy/effectiveness” (SD 0.7) and

     • “Q3, relevance and validity of evidence” (SD 0.6).

   • Largest variations in weights across the group were observed for the criteria:

     • “D2, size of population” (SD 1.2) and

     • “Q1, adherence to requirements of decision-making body” (SD 1.3).

   • Participants found the criteria most important to decisions:

     • “Q3, relevance and validity of evidence” and

     • “I1, improvement of efficacy/effectiveness” with mean weights of 4.7 and 4.5, respectively.

   • The lowest weights were assigned to the criteria:

     • “C1, clinical guidelines” (3.0 [SD 1.0]) and

     • “E1, budget impact” (3.0 [SD 0.9]).

Figure 1: Normalized Weight Scoring in the Scoring System of Goetghebeur et al.

Note: The cumulative value of all 'normalized weights' is 1.00. The proposed weighted ratios may be readjusted as needed for the revised MCDA matrix. For the proposed MCDA core model, an estimate close to 1 would represent an ideal intervention. Consistent use of the MCDA model and applicable methodology by the same decision-making committee will provide more accurate value estimates for ranking and comparing interventions within a contextual framework.

1. Iskrov et al., 2016²¹

   • The highest weight was given to criteria describing health technology's characteristics (44 points), followed by those describing the indicated disorder (32 points) and public health considerations (24 points). Lifesaving was considered the most important criterion within the health technology category​.

   • Criteria such as disease severity and disease burden were critical, with more emphasis on chronic, life-threatening conditions. Stakeholders recognized the need for real-world data and the impact of rarity on orphan drugs' evidence.

2. Jakab et al., 2020¹⁴

   • The final outcome of this process was a set of 16 general criteria proposed for local adaptation in MCDA frameworks. This set was not presented in a ranked order but as a comprehensive pool from which stakeholders can select and adapt criteria based on local priorities and context; a weighting assessment/proposal was not carried out.

3. Zelei et al., 2021²³

   • Three highly referenced value frameworks were selected (i.e., the International Society for Pharmacoeconomics and Outcomes Research [ISPOR] value “flower,” the ICER value framework, and the Second Panel on Cost-Effectiveness impact inventory) to categorize criteria into 3 groups according to their use in traditional HTA. The terminology of the ISPOR value flower was used: (1) core criteria (i.e., criteria using traditional HTA), (2) standard but inconsistently included criteria (i.e., criteria assessed in certain settings or jurisdictions), and (3) novel criteria (value criteria typically not assessed in traditional HTA.

   • Summarized in terms of the most frequent core value criteria (type and frequency of inclusion):

     • comparative clinical effectiveness

     • comparative safety and adverse events and side effects

     • health-related quality of life and PRO effects

     • longevity effects.

   • Summary of the most frequent common value criteria (type and frequency of inclusion):

     • adherence-improving factors (posology, patient convenience, comfort).

   • Summary of the most frequent novel value criteria (type and frequency of inclusion):

     • unmet medical needs and availability of alternative treatments

     • severity of the disease

     • reduction in uncertainty (strength, quality, level of evidence)

     • type of therapeutic benefit (curative, lifesaving)

     • size of population/prevalence

     • burden of illness

     • level of innovation

     • equity and support vulnerable groups

     • manufacturing complexity

     • expert opinion

     • production of toxic waste pollution by intervention.

Appendix 3: MCDA Model of Goetghebeur et al., 2012²⁰

Note that this appendix has not been copy-edited.

Figure 2: Decision Criteria Included in the MCDA Model of Goetghebeur et al.