CADTH Health Technology Review

Deliberative Appraisal Processes in Health Technology Assessment

Environmental Scan

Authors: Marina Richardson, Nicole Mittmann, David Kaunelis

Key Messages

• Deliberation — the bringing together of individuals with multiple perspectives, experiences, and values to critically examine evidence, share reasons and rationales, and reach a decision on a course of action — occurs at all stages of the health technology assessment (HTA) process.

• The focus of this report is on deliberation at the committee appraisal stage of HTA.

• The objectives are to review and conceptualize the evidence and key considerations for deliberative processes at the committee appraisal stage of HTA and the available guidance for assessing the quality of the deliberative process at this stage.

• National and international practices for deliberative processes at the committee appraisal stage of HTA vary depending on contextual considerations (e.g., type of health technology and the medical condition being treated), the mandate of the HTA organization, health system and funding structures, and societal values.

• Each aspect of the deliberative appraisal process should be rooted in the values of the health system and be principled, clear, transparent, purposeful, and reasoned.

Summary

• Deliberation during the health technology assessment (HTA) process involves bringing together individuals with multiple perspectives, experiences, and values to critically examine evidence around a health technology, share reasons and rationales, and reach a decision on a course of action.

• Deliberation may occur across all stages of the HTA process (e.g., scoping, assessment, appraisal, and implementation and monitoring). The focus of this review is on deliberation at the committee appraisal stage; however, the methods used at other stages of the HTA process may impact processes used at the appraisal stage.

• The objectives of this report are to:

  • review and conceptualize the evidence on deliberative processes for the committee appraisal stage of HTA

  • review and summarize key considerations for deliberative processes at the committee appraisal stage of HTA

  • review the available guidance for assessing the quality of the deliberative process at the committee appraisal stage of HTA.

• The committee deliberative appraisal process is defined here according to the input-throughput-output model.

• Input into the deliberative appraisal process is typically tailored to the needs of decision-makers and comprised of either a full or partial consideration of the components of a full HTA review.

• Throughput considerations within the deliberative appraisal process that are predominantly discussed in the literature and in practice include extent of stakeholder involvement, appraisal committee size and structure, the role of the committee chair, the structure of the deliberative process, managing uncertainty, openness of the deliberation, and arriving at a recommendation or decision.

• Output of the deliberative appraisal process aims to meet the needs of stakeholders and allow for opportunities for stakeholder feedback.

• Other considerations shaping deliberative appraisal processes in HTA include the life-cycle approach to HTA (e.g., considerations for reassessments), real-world evidence (e.g., using and communicating the use of real-world evidence), responding to the need for virtual meetings, efficiency and sustainability of expert appraisal committees, and the fluidity of the deliberative appraisal process.

• National and international practices for deliberative processes at the committee appraisal stage of HTA vary. Processes are context-specific and there is no one-size-fits-all approach.

• Each aspect of the deliberative appraisal process should be rooted in the values of the health system and be principled, clear, transparent, purposeful, and reasoned.

• This review was informed predominantly by HTA-related literature; however, future research that considers literature and expertise from other disciplines that involve a deliberative appraisal step with an expert committee are likely to provide valuable insights for HTA.

Context

Heath technology assessment (HTA) is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points over the course of its life cycle.¹ The purpose of HTA is to inform decision-making to promote an equitable, efficient, and high-quality health system.¹ The HTA process consists of several steps, including scoping, assessment, contextualization (or appraisal), and implementation and monitoring.² The dimensions of value considered in the HTA process may include clinical effectiveness, safety, costs and economic implications, and ethical, social, cultural, legal, organizational, and environmental considerations.¹

The contextualization or appraisal step of the HTA process typically involves an expert review committee. The evidence informing each of the relevant dimensions of value are brought forward to this committee within a deliberative process that involves “the critical examination of an issue” and “the weighing of reasons for and against a course of action.”³ The appraisal step is a critical component of the HTA process that facilitates bringing the scientific, contextual, and experiential forms of evidence together with individuals with multiple perspectives, experiences, and values. Value judgments occur throughout the entire HTA process⁴; for example, during the assessment stage with the selection of outcomes, types of evidence considered, and cost-effectiveness modelling assumptions or during the appraisal stage with deciding who will be involved, how they will be involved, and how the relative importance of the factors under consideration are perceived.⁴ Given the different values at stake, it is unlikely that all stakeholders involved will agree on a decision; therefore, acceptance of the process becomes important.⁵

In the context of HTA, there is a lack of consensus about what is meant by the process of deliberation or deliberative processes. An evidence-informed deliberative process framework has been developed to support HTA agencies in making reimbursement recommendations by combining a structured decision-making process with a process for making choices between policy options that reflect stakeholder perspectives.⁶ This definition of deliberative processes is broad because it covers the complete HTA process (i.e., scoping, assessment, appraisal, and implementation and monitoring) and identifies how HTA agencies “should ideally organize their processes to achieve legitimate decision-making.”⁷ A narrower scope and related definition of the deliberative process were outlined in the publications from the Health Technology Assessment International (HTAi) 2020 Global Policy Forum, which contextualized the deliberative process as an input-throughput-output (ITO) model focusing on the appraisal of pharmaceuticals. Input describes the material that forms the basis of the deliberation, throughput describes the process of the deliberation, and output refers to how the content and outcome of the deliberation are communicated.³ Several existing frameworks guide the development of evidence-informed deliberative processes and the ITO model, including accountability for reasonableness and multicriteria decision analysis (MCDA).

Increasing pressures for timeliness, transparency, and patient and public engagement, as well as an evolution in the types of health technologies in development (e.g., interventions for rare diseases), treatment paradigms (e.g., tumour-agnostic drugs), and types of evidence considered (e.g., basket trials, single-arm trials, patient and clinician input), are adding pressures to — and increasing the importance of — the deliberative appraisal stage of HTA. There is recognition that differences in processes between jurisdictions are warranted considering the links between institutional priorities and practices, and societal needs and preferences for treatment.⁸ However, it is unclear which aspects of the deliberative process are particularly influential, how they may influence the output of committee deliberations, and if there is guidance on best practices.

The objective of this work was to conduct an Environmental Scan to examine the deliberative processes within the appraisal phase of HTA to help inform a consistent, credible, and transparent process in the Canadian context.

Objectives

The key objectives of this Environmental Scan are as follows:

• Objective 1: Review and conceptualize the evidence on deliberative processes for the committee appraisal stage of HTA. How do we define deliberative processes in HTA and what are the component parts?

• Objective 2: Review and summarize key considerations for deliberative processes at the committee appraisal stage of HTA. What aspects of deliberative processes at the committee appraisal stage are considered in the context of HTA?

• Objective 3: Review available guidance for assessing the quality of the deliberative process at the committee appraisal stage of HTA. What guidance is available to assess the quality of deliberative processes at the committee appraisal stage of HTA?

Methods

This Environmental Scan is informed by a limited literature search and through discussions with internal and external experts and CADTH staff involved in national or international HTA activities. Relevant published literature was identified primarily through targeted MEDLINE and Embase searches. Grey literature was retrieved through a focused internet search, primarily through the websites of Canadian and international HTA agencies. Additional studies and reports were identified from the reference lists of included articles and consultations with content experts in Canada and internationally between September 2019 and August 2020. The literature search was conducted by an information specialist and limited to English-language documents published between January 1, 2014, and September 6, 2019, with monthly alerts up to December 1, 2020. Conference abstracts were excluded. The search strategy comprised both controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings), and keywords. The main search concepts were deliberative processes and/or frameworks and HTA (including names of specific HTA organizations). This analysis focused on a limited number of countries with well-established HTA practices and appraisal committees: Canada, France, the US, the UK, Scotland, Australia, the Netherlands, and Germany.

Findings

A total of 403 studies were identified by the literature search. Due to the nature of this report and the overlapping concepts between research objectives, a specific number of studies identified for each research question is not stated. The findings presented are based on the results of the literature search and discussions with content experts.

Conceptualization of Evidence Informing HTA Deliberative Appraisal Processes

Although there is a lack of consensus about what is meant by the process of deliberation or deliberative processes in HTA, the accountability for reasonableness framework, core principles (transparency, impartiality, and inclusivity), models (evidence-informed deliberative process and ITO), and methodological frameworks for assessing (e.g., European Network for Health Technology Assessment [EUnetHTA] Core Model, INTEGRATE-HTA), appraising (e.g., MCDA), and decision-making (e.g., GRADE Evidence to Decision Framework [EtD]) exist to guide the HTA process. Figure 1 illustrates the relationships among these elements using the steps of the HTA process (i.e., scoping, assessment, appraisal, and implementation/monitoring)² as an anchor. The examples mentioned are not meant to indicate that they should be used, but to provide examples of existing frameworks and illustrate how they are intended to function within the context of the HTA process.

Accountability for reasonableness is an ethical framework that outlines 4 criteria — publicity, relevance, revision, and enforcement — that when satisfied aim to ensure that the resource allocation process is fair and just.^5,9

In addition, a set of guiding principles that could be used to support the development and management of a well-functioning deliberative process in HTA for the phase of appraising pharmaceuticals was identified at the 2020 HTAi Global Policy Forum. Three principles were identified as aspirational principles for all agencies and stakeholders to strive for: transparency, impartiality, and inclusivity. Other principles identified at the forum included consistency, reasonableness, reviewability, responsibility, and accountability.¹⁰

Figure 1: Conceptualization of the Evidence Informing HTA Processes for Reimbursement or Coverage Recommendations

EDP = evidence-informed deliberative process; EtD = Evidence to Decision; EUnetHTA = European Network for Health Technology Assessment; HTA = health technology assessment; ITO = Input-Throughput-Output; MCDA = multicriteria decision analysis.

Note: The focus of this Environmental Scan is on the contextualization part of the HTA process (defined using the ITO model).

Source: ITO model,³ HTA process (scoping, assessment, contextualization, implementation, and monitoring).²

The EUnetHTA Core Model, INTEGRATE-HTA, GRADE EtD framework, MCDA, and evidence-informed deliberative processes, are examples of frameworks that vary in scope in terms of their intended use (e.g., to support assessment, appraisal, or the entire HTA process), criteria considered (e.g., clinical, economic, ethical), and the extent to which decision rules (e.g., thresholds for evidence) or deliberative guidance (e.g., extent of stakeholder involvement) are included. The EUnetHTA Core Model was developed to provide a framework for producing and sharing HTA information.¹¹ The model is intended to provide a common framework for conducting and reporting HTA information internationally. Other assessment frameworks and methods, such as INTEGRATE-HTA and Methods for Health Technology Assessment of Medical Devices: A European Perspective (MedtechHTA), recognize the need for methods that capture the nuances of complex technologies and medical devices, respectively. INTEGRATE-HTA is a 5-step process for a comprehensive, integrated assessment of a complex technology that is patient-centred, comprehensive, and takes context and implementation into account.¹² MedtechHTA provided recommendations to build on the HTA process to accommodate medical devices as complex health technologies and, in particular, consider the implications of the learning curve associated with medical devices, incremental innovation, dynamic pricing, and organizational impact.¹³

Decision frameworks are used to guide decision-making about whether or not to pay or partly pay for a new health technology.¹⁴ For example, GRADE EtD frameworks have evolved from the GRADE approach used to assess the strength of recommendations for health technologies.¹⁵ EtD frameworks have been developed to support clinical practice guidelines, coverage decisions, and health system and public health recommendations or decisions.¹⁵ MCDA aims to inform the decision-making process in a structured way.¹⁶ MCDA involves 3 steps: defining the decision problem, selecting criteria that reflect relevant values, and constructing the performance matrix. MCDA may be defined as taking the form of qualitative MCDA, quantitative MCDA, or an MCDA with decision rules.¹⁶ The evidence-informed deliberative process is a framework that aims to support HTA agencies at different steps of the HTA process by combining a structured decision-making process with a process for making choices between policy options that reflect stakeholder perspectives.¹⁷ The ITO model is a general evaluation framework and was used to capture considerations for the deliberative appraisal of pharmaceutical products with overlap on assessment and implementation and monitoring steps of the HTA process.

The information that is used to inform the appraisal of an HTA are typically defined or described as assessment or value assessment frameworks, deliberative frameworks, or a list of evidentiary domains that are considered during the deliberative appraisal. For example, value assessment frameworks identify the criteria and their relative importance to assess the value of a health technology.¹⁸ However, the intended use of value assessment frameworks vary between contexts. The American Society of Clinical Oncology (ASCO) framework was developed in the context of patient-clinician conversations, the Institute for Clinical and Economic Review (ICER) framework was developed to support value-based prices for health technologies, and the HTA Core Model was intended for use in assessing technologies. It is recognized that definitions are needed for the core components of a framework for assessing the value of a health technology, and that frameworks should be designed to follow principles such as transparency, predictability, accountability, and consistency.¹⁸ The approach to defining value varies according to the culture and values of a country, the institutional context, and the organization of the health care system.¹⁸

The focus of this Environmental Scan is related to the appraisal part of the HTA process using the ITO model described by Bond and colleagues¹⁰ (the elements highlighted in green in Figure 1).

Considerations for Deliberative Appraisal Processes Using the ITO Model

The focus here is on the throughput and output components of the ITO model. However, the input step influences the throughout and output components and, as such, these considerations are also included. The importance of context that may moderate the evidentiary and process aspects of the deliberative appraisal will be discussed first followed by a discussion of considerations based on the ITO model.

Country and HTA Agency Mandates

Internationally, countries have national and/or regional HTA organizations that employ committees to deliberate on evidence and other relevant information regarding health technology. In each country, the decision-making process varies and the legislative authority of the recommendation or decision-making committees differ. For example, in Canada, non-binding recommendations are made by national HTA committees to individual provincial (except for the province of Quebec), territorial, and federal decision-makers. However, in the UK and Germany, recommendations made by national review committees are binding for decision-makers. These country-specific factors may influence the deliberative appraisal process used by the HTA body (e.g., information considered and perspective taken). Countries also vary regarding different committees for drug and non-drug technologies (Table 1). In the US, for example, ICER does not have separate committees for drugs and medical devices and uses the same value assessment framework to evaluate all technologies, such as drugs, devices, tests, and delivery system innovation. However, judgments regarding evidence or determining thresholds for budget impact depend on the nature of the health technology under consideration.¹⁹ For Canada, the UK, Scotland, and Australia, separate committees exist for drug and non-drug technologies and, in some cases, there are separate committees for prostheses (Australia) and diagnostics (UK). Countries may also have different budgets designated for different types of technologies, such as the Cancer Drugs Fund in the UK.

Country-specific social values may also influence the deliberative appraisal process in HTA. Social values have been defined as distinct from moral or ethical values in that the former give shape to universal moral values based on the social, cultural, religious, and institutional features of a given society.²⁰ Whitty and Littlejohns (2015) defined social values as “those values that are considered important by the society that is affected by the decision-making process.”²¹ Clark and Weale (2012) provided a conceptual framework for assessing social values in a health priority setting, identifying processes (i.e., how decisions are made) and content values (i.e., what decisions are made and why).²⁰ NICE has a social value principles document that provides the foundational guidance for NICE advice and decision-making.²² In Ontario, Canada, an interactive participatory approach is used to elicit ethical and social values to inform the HTA review and appraisal process.²³ Top ethical values identified include ensuring that individual autonomy is respected, protecting the physician-patient relationship, demonstrating respect for the conception of certain persons, and ensuring that human dignity is respected.²³ Top social values identified include demonstrating respect for religious, social, and cultural convictions; ensuring that technologies are aligned with social values and arrangements; evaluating how the technologies relate to more general challenges of modern medicine; and evaluating how the technologies affect distribution of health care.²³ Studies to assess social values in HTA have been conducted in the context of specific disease areas, such as hepatitis C in Canada.²⁴

Type of Health Technology

HTA is used to support decision-making for a range of health technologies. Drugs and medical devices are 2 common categories used to classify health technologies, although it is recognized that there are important differences between the 2 that should be considered when assessing value. Often the cost-effectiveness question for devices and drugs differ; for example, optimization of cost-effectiveness is typically of primary concern for drugs, whereas decisions about procurement and/or incorporation of the device in practice²⁵ are often of primary concern for medical devices. In the context of conducting economic evaluations, Drummond et al. (2009)²⁶ highlighted 6 key differences between drugs and medical devices that should be considered: devices may be diagnostic as well as therapeutic, devices and drugs have different types of clinical evidence (e.g., randomized controlled trials [RCTs] are often difficult, and blinding is challenging for medical devices), efficacy of the device often also depends on physician skill, implementation often has wide implications for medical devices, there is often a lack of evidence for all procedures in the same class, and prices often change over time due to product evolution or process of procurement. Devices and drugs are also regulated, funded, appraised, and implemented through different pathways. Other health technologies, such as non-pharmaceutical curative health technologies (e.g., surgery), non-curative health technologies (e.g., psychotherapy), and/or social care, have less commonly been subject to HTA.²⁷ A study conducted in the Netherlands contrasted these less commonly assessed health technologies with outpatient pharmaceuticals, citing several characteristics that make outpatient pharmaceuticals particularly suitable for HTA, including the existence of marketing authorization, an identifiable and accountable counterparty, and the product characteristics.²⁷

Even within certain classes of drugs (e.g., orphan drugs, regenerative medicines) it has been argued that different HTA processes are needed to appropriately assess value.^28-31 Nicod et al. suggested supplemental processes for rare diseases across the full HTA process. Recommendations specifically for the deliberative appraisal committee process included having committee members with rare disease expertise, broader considerations of value, greater leniency in the quality of the evidence, and more flexibility in the economic evidence and including different decision rules (e.g., different cost-effectiveness threshold).²⁸ In the context of regenerative medicines and cell therapy products, 1 study in the UK used a mock deliberation of a panel convened by NICE and found that methods to handle the uncertainty in the evidence and the evidentiary challenges were not unique to regenerative medicines (e.g., small studies, lack of long-term data, variable clinical efficacy).³² The study found that the deliberative process in particular already accounted for “the desirability of promoting innovation” by considering higher cost-effectiveness thresholds if the health technology was deemed innovative.³² However, the area where there was opportunity for changing the deliberative process for certain technologies, such as regenerative medicine, related to the need for additional analyses to demonstrate the impact of various pricing schemes or to inform an understanding of the potential health benefit achieved by immediate approval versus waiting for future research to be undertaken.³²

Type of Medical Condition

The severity of the disease, rarity of the disease, and availability of alternatives are other contextual considerations that may influence the methods used to assess value. Similar to ICER’s consideration of the type of health technology, ICER also considers modifications to its methods for assessing value for treatments of serious, ultra-rare disorders and high-impact single-administration or short-term therapies. Most often, HTA agencies make recommendations for rare disease drugs based on the same centralized national review process as other drugs³³; however, several HTA bodies have created separate mechanisms, adapted existing processes,^29,33 and/or added “safety nets” that allow for individual patient requests for access to medicines that do not receive a positive reimbursement requests.³³ These alternatives have arisen primarily because orphan drugs are not able to meet typical cost-effectiveness standards, including high costs and uncertainty in the clinical evidence.^34,35 A study of 32 countries (mostly European) found that 41% of countries had a supplemental process for rare or ultra-rare disease treatments, and the countries were classified based on the extent to which these processes were integrated with standard HTA processes.²⁸

In Canada, separate committees exist at CADTH to evaluate oncology and non-oncology drugs, and decision-makers include both ministry drug plans and cancer agencies depending on the jurisdiction. Australia’s Pharmaceutical Benefits Advisory Committee (PBAC), the Dutch National Health Care Institute (ZIN), NICE, and France’s Haute Autorité de santé (HAS) do not have HTA criteria or committees specific for oncology or orphan diseases. However, NICE has a Highly Specialised Technology evaluation committee that exists for ultra-orphan drugs.³⁶ PBAC and NICE do not have separate budgets for orphan drugs; however, there are options for funding orphan drugs (through the life-saving drug program) or oncology orphan drugs through the Cancer Drugs Fund (UK).³⁶ The UK’s Cancer Drugs Fund was established in 2010 and reformed in 2016 to become a managed access scheme. The existence of the fund is designed to provide patients with earlier access to drugs, to resolve uncertainty in clinical evidence, and to consider off-label indications. There was early criticism for the drug fund, including questioning the underlying ethical basis for the existence of the fund.³⁷ Funding cancer drugs that have an incremental cost-effectiveness ratio greater than other drugs in the health system result in a disproportionate loss of health benefits to other parts of the National Health Service.³⁷

Other Contextual Factors

There are several additional contextual factors (i.e., situation-specific factors) that may moderate the evidentiary and process aspects of the deliberative appraisal process (e.g., tolerance for uncertainty in clinical or economic evidence). In Western Europe, a list of contextual factors that are considered in health care coverage decisions include nature of the illness, considerations for equity, costs to individuals, responsibility of individuals, medical necessity, severity of the condition, need, absence of alternatives, impacts beyond the patient, contribution to societal functioning, compassion to vulnerable groups, and alternative health technologies used if not covered.³⁸ In the US, ICER considers similar contextual considerations in their value framework and committee deliberations, including a similar mechanism of action to other treatments, considerations for historically disadvantaged or underserved communities, health loss without treatment based on an absolute and proportional quality-adjusted life-year (QALY) shortfall, impact on family and caregivers, and impact on improving return to work or productivity. ICER also considers uncertainty in evidence, delivery mechanism or complexity of intervention, and timing of benefits and risks.

Input Step of ITO Model

Bond et al. (2020)¹⁰ identified the input step of the ITO model for the deliberative process to be the step where evidence, information, and perspectives are collected. The type and extent of evidence, information, and perspectives collected during the input stage is typically variable across technologies depending on the contextual considerations as described previously (e.g., country, HTA agency mandate, health technology, medical condition, and other considerations). Table 5 (Appendix 1) outlines a set of criteria considered in selected assessment and decision frameworks. HTA agencies also typically have a list of criteria considered during the deliberative appraisal step. Although the input and throughput steps of the ITO model are described separately here, the input (i.e., assessment) step informs the throughput (i.e., appraisal) step, and it is often not well-described for stakeholders how these 2 steps are linked or operationalized at a deliberative appraisal committee meeting.

In this review, 8 general domains have been identified that are typically considered when assessing the value of a health technology: patient and caregiver perspectives; clinical evidence; economic evidence; and implementation, legal, ethical, social, and environmental considerations. Although these domains have been classified separately, there is overlap between the domains, many domains contain several subdomains, and terminology for these domains and subdomains may vary across jurisdictions and contexts. The EUnetHTA Core Model provides a comprehensive overview and consideration of the potential overlap among criteria considered within HTA.³⁹ Given the detailed discussions of the assessment criteria in other publications (e.g., EUnetHTA documents), the discussion here is limited to an outline of common elements discussed within each of these 8 domains as presented in Appendix 2. We have also used these 8 criteria to categorize the decision criteria for resource allocation and health care decision-making that was systematically identified in 2012 by Guindo et al.⁴⁰ and the determinant criteria for basic health benefit packages in health systems around the world in the scoping review conduced by Hayati et al. in 2018⁴¹ (Table 6, Appendix 1). The criteria presented in Table 6 are not an exhaustive list nor do they apply to all countries, contexts, or technologies. Furthermore, not all elements identified will be included in a written HTA assessment report but may arise during expert committee deliberations. Explicitly stating the domains and subdomains considered during deliberative processes is important to not overlook any criteria⁴² and to provide consistency, transparency, and predictability for submitters and committee members. However, explicit statements of domains and subdomains listed should not be considered exhaustive or restrictive.⁶ How these elements are operationalized during deliberations is discussed in the Throughput section.

Throughput Step of ITO Model

Bond et al.¹⁰ identified the throughput step of the ITO model for the deliberative process as the step where evidence and information are critically examined and reasoned. A comprehensive list of considerations in the throughput stage of the ITO model was described in the HTAi background paper for the 2020 Global Policy Forum meeting.³ The focus of this review will be on stakeholder involvement; committee size, expertise, and structure; the role of the committee chair; the structure of the deliberation and decision rules; managing uncertainty; openness of committee meetings; and arriving at a decision or recommendation. There are several aspects of the deliberative appraisal process that likely influence the coverage or reimbursement recommendation that have received much less attention,⁴³ and some are discussed here. Although this review does not include an exhaustive list of all aspects of the deliberative process, it does provide a foundation of key elements for consideration.

Stakeholder Involvement

Stakeholders are defined as those individuals or groups of individuals who may be potentially impacted by a decision.⁴⁴ Stakeholder involvement throughout the HTA process is a critical element of a fair and legitimate decision-making process.⁴⁴ The focus here is on stakeholder involvement during the deliberative appraisal stage (i.e., throughput part) of the process. However, involvement that occurs at other phases of the HTA process may influence the extent of involvement at the appraisal and decision-making stage. Stakeholder involvement has been defined here in the broadest sense; that is, it may vary from providing information (lowest level of engagement) to engagement (where there is an emphasis on shared understanding of perspectives and the potential impacts of decisions) and partnering (highest level of engagement) where there is patient, citizen, or other relevant stakeholder empowerment.⁴⁵ Stakeholder involvement may occur at the scoping phase, the assessment stage (e.g., written input to inform protocol development for the systematic review or feedback on a draft evidence report), as part of the committee during the appraisal stage (e.g., as an expert committee member or invited participant), or as responding to draft recommendations or reports. Stakeholders may be involved in identifying evidence production for HTA,⁴⁶ or their involvement may come after the completion of the HTA process. The 7Ps Framework to Identify Stakeholders in Patient-Centred Outcomes Research and Comparative Effectiveness Research developed by Concannon and colleagues⁴⁷ identified patients and the public, providers, purchasers, payers, policy-makers, product makers (i.e., manufacturers), and principal investigators (i.e., researchers). The focus of the discussion here is on patients, the public, providers, payers, and product makers.

Patient and Public Involvement

Abelson and colleagues⁴⁸ describe 6 principles to guide patient and public involvement in HTA: their involvement should be purposeful, pragmatic, fair and equitable, proportional, evidence-informed, and transparent. Depending on the stage of the HTA process, the form of involvement of patients and the public will vary. Important elements in the deliberative appraisal process are clearly articulating the principles and goals of public and patient involvement, establishing a common language to support involvement, and measuring and evaluating involvement to assess if the goals of involvement are being achieved.⁴⁸

An important distinction in patient and public involvement in the HTA process is the difference between the role of patients and the role of the public. Street et al.⁴⁹ emphasize that patients and the public have different interests that should be considered in the HTA process (including in the deliberative appraisal process) and clearly articulate definitions and rationales for their inclusion. For example, the public is defined as “a community member who holds the public interest and has no commercial, personal, or professional interest in the HTA process,” whereas a patient is defined as “an individual with a disease or disorder who is using some aspect(s) of the health care system because of this disease or disorder.” Furthermore, a consumer is similar to, but distinct from, a patient in that they are an individual who “uses, has used, or intends to use a particular technology or service.”

Patient and public involvement in the deliberative process can take many forms; for example, some jurisdictions have open meetings where patients and the public can register for time to speak during committee meetings, and other jurisdictions seek input directly from patients and the public (and/or from the literature) in advance of the meeting which is brought forward by the HTA assessment teams or the committee members during the meeting. In Canada, for example, the opportunity for patient groups to submit input to help contextualize and inform the review of pharmaceutical products is a form of patient involvement that is considered by the review team in their interpretation of the evidence, and it is also brought forward to the expert committee during deliberations. Other avenues for patient and public involvement are eliciting input from patients and the public on draft documents to be presented to the committee, eliciting input on draft recommendations after the meeting, or including patient and public members as part of the expert committee. Individuals selected to be patient or public members as part of a committee should be clearly defined. Street and colleagues also offer clear definitions for committee members: those who hold the role of public, patient, and consumer members are “individuals selected to support the inclusion of the interests” of society at large, patients, and consumers, respectively, on the committee.⁴⁹

Patient and public involvement in the appraisal process may also come via indirect means, such as through elicitation of social values to inform the process and the criteria (i.e., content) considered during the deliberative process.²⁰ NICE, for example, has a Social Values Framework that underlies their approach for developing guidance and helps the committee resolve uncertainty in evidence.²² Other countries (e.g., Australia^21,50,51 and some in Eastern Europe⁵²) have sought public or societal input on aspects of their decision-making process. Although there are some similarities between jurisdictions, social values are likely context dependent. In Canada, some of this work has been done to inform HTA deliberative processes in Ontario⁵³ and to inform cancer care.⁵⁴ The focus here is on the deliberative appraisal aspect of HTA; however, Abelson (2018) emphasized the importance of considering patient and public engagement holistically across the entire HTA process with a clear organizational-level vision to ensure meaningful engagement.⁵⁵

Providers, Product Makers, and Payer Involvement

The extent of product maker (or manufacturer) involvement also depends on the submission process used in a particular jurisdiction and the health technology type. HTA agencies that initiate technology reviews based on a prioritization process may request submissions from industry (e.g., NICE), and stakeholder input throughout the assessment process is often accepted from patient groups, health care providers, payers, the manufacturer of the health technology, as well as from comparator technology manufacturers. For manufacturer-initiated technology review processes (e.g., drug reviews in Canada), the list of eligible stakeholders is typically more restricted. Input from health care providers, product makers, and payers may come throughout the review process or during expert committee deliberations. These stakeholder groups may be provided with an opportunity to provide additional perspectives on the technology under review in addition to the expertise provided by the permanent committee membership as described below. Input from a payer perspective may allow for input on pricing, access, affordability, and coverage policies, and may help identify questions about treatments of interest, comparators, and key outcomes.¹⁹ Ireland, for example, has a Rare Diseases Technology Review Committee that provides input on managed entry agreement proposals that takes into consideration clinician and other stakeholder input.²⁸

There is limited evidence available to assess how providers, product makers, and payers are best engaged in the deliberative appraisal process. Appropriate management of conflicts of interest³ for all stakeholder groups is particularly important for achieving a fair and impartial process.

Committee Size, Expertise, and Structure

Unless there is strong consensus among a group of experts on a particular topic, expert committee composition and size has an impact on the outcome of the deliberative process.⁵⁶ Empirical literature assessing the impact of committee size and structure specifically in the context of HTA is limited; however, insights can be gained from other areas of study. Regarding medical device purchasing decisions, Rosina et al. (2014) reported that the size of the panel, expertise of the panellists, and consistency of the panel were key factors that impacted final decisions.²⁵ For example, a small panel may not take all details into consideration, whereas a large panel may result in an inability to achieve consensus,²⁵ or fewer opportunities for individual committee members to be heard.⁵⁷ Larger panels can also increase cost,²⁵ especially in situations where in-person attendance at meetings is common and/or the frequency and length of reviews requires considerable preparation and review time by committee members. Committee size typically ranges between 10 and 20 members⁵⁷; more than 20 members is often not practical.⁵⁶ However, there are several examples of advisory committees that have more than 20 members, such as HAS (29 members), NICE (24 members), and the Scottish Medicines Consortium (SMC) (23 members).³

Committee member expertise should include considerations for technical knowledge, professional experience, and diversity (demographically and professionally)²⁵ and may include individuals that represent public, private, or community perspectives. The full committee composition should be appropriate to bring forward the issues related to the health technology under consideration, including potential issues related to the feasibility or acceptability of the policy.⁵⁷ Morestin and colleagues emphasized that deliberative processes cannot be described as representative given the constraints of the number, expertise, and perspectives of the individuals at the table, and the time available for each deliberative discussion.⁵⁷ Committee membership may also be complemented by additional health care providers, patients, the public, or other individuals who may offer expertise as advisors depending on the technology under consideration.

The composition of committees varies across jurisdictions. Health care providers are consistently included across national and international HTA expert committees. Type of clinical expertise may vary between type of committee; for example, the Medical Device and Health Technology Evaluation Committee at HAS includes surgeons and an occupational therapist.⁵⁸ Inclusion of other types of members (e.g., public and patient members, payers, industry, economists, ethicists, lawyers, and engineers) are more variable. For example, patient and public members are included at NICE, and public partners are included at the SMC in Scotland. Scotland, and Australia have industry representatives on (some) their committees. It has been suggested that having an industry perspective in deliberative committees can help to strengthen the knowledge and diversity of committees and potentially increase the acceptability the industry of the process and the resulting decisions.³ The Prostheses List Advisory Committee in Australia includes advisory members who represent the medical device industry, private health insurance, and private hospitals. There are also manufacturer representatives on the drug expert committees in Scotland and at NICE.⁵⁹ Payer representatives are included on SMC committees, whereas CADTH includes payers as observers at drug committee meetings. Committee composition may also depend on the mandate of the committee. For example, in the Netherlands, 2 types of committees exist: a scientific advisory committee that considers the scientific evidence base underlying the HTA report and an appraisal committee that is appointed by the crown and intended to represent society (and includes a patient member and a medical ethicist) which considers factors such as the balance between the cost-effectiveness and burden of disease.

Role of the Committee Chair

The committee chair or facilitator has an important role in managing committee meetings,^3,43 and their importance has been recognized as an under-appreciated aspect of the HTA process.⁴³ During the deliberation, perspectives must be elicited and aggregated,⁵⁶ clarified,⁵⁷ and managed in terms of who is speaking, which often requires prompting or curtailing discussion.⁵⁶ A committee chair should use neutral language¹⁰ and present the problem or ask questions in an unbiased manner.⁵⁷ An action item that HTA agencies can implement that aligns with the principle of impartiality (i.e., the deliberative process being free from undue influence) is to have a committee chair who manages the deliberation in a way that allows committee members to have equitable input and prevents their own opinions from moving deliberations forward.¹⁰ There is limited empirical literature available in the context of HTA to understand the influence and best practices for the role of the committee chair during deliberations.

Structured Deliberative Processes

The development of HTA recommendations for reimbursement decisions involves a complex weighing of evidence from multiple disciplines and perspectives. The consistency and transparency of recommendations from HTA agencies are often criticized; consequently, HTA agencies have developed independent processes to manage this complexity. MCDA is a means to enhance consistency and transparency by identifying, collecting, and structuring information to support the deliberative process,⁶⁰ and it allows for a structured way to include different societal values.¹⁶ The traditional form of MCDA involves 3 steps: defining the decision problem, selecting criteria that reflect relevant values, and constructing the performance matrix.¹⁶ Baltussen (2019)¹⁶ described 3 types of MCDA: qualitative MCDA, quantitative MCDA, and MCDA with decision rules depending on how the HTA process operationalizes the performance matrix. In qualitative MCDA, the committee makes a qualitative interpretation of the performance matrix; in quantitative MCDA (or “full” MCDA), stakeholder preferences regarding the relative importance of criteria are measured using criterion weights; in MCDA with decision rules, trade-offs between the criteria in the matrix are made using a set of rules that can be qualitative or quantitative in nature.¹⁶ The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) MCDA Emerging Good Practices Task Force also suggests that not all MCDA steps are required in deliberative processes.^60,61

MCDA processes can enhance the consistency and transparency of the deliberative process but are limited by challenges such as how criteria are defined (i.e., by whom, if they are fixed) and weighted (based on whose preferences), how to consider opportunity costs, and how to address uncertainty.⁶² Quantitative weighting of criteria has been found to require substantial investment in identifying weights and may not always have appropriate societal representation.⁶³ There is also likely to be variation between technologies and practical challenges of engaging individuals with sufficient knowledge of the MCDA process.⁶³ The use of quantitative MCDA with the appraisal committee in the Netherlands (ZIN) was assessed in a panel meeting and it was found that quantitative MCDA prevented the richness of the argumentation seen with non-quantitative deliberations, but the committee appreciated the explicit deliberation of the relative importance of each criterion.⁶⁴ The committee felt that the structure allowed for arguments to be voiced, showed where there was consensus, and allowed the committee to focus discussions on the elements that mattered most to stakeholders.⁶⁴ Baltussen and colleagues suggested that HTA agencies should use explicit criteria in their recommendations (e.g., at a minimum qualitative MCDA) and stated that MCDA with decision rules is essentially akin to the “structured deliberation” present in the Netherlands and the UK.¹⁶ The evolution in the use and description of the typology used to describe MCDA has also led to a recognition that deliberation is a key commonality across all approaches. DiStefano and Krubiner⁶⁵ emphasized that future research should focus on how best to facilitate a deliberative environment that “enables a rich discussion of trade-offs” instead of trying to identify tools that attempt to quantify or oversimplify decision-making.

Another framework that has emerged to support more structured decision-making in HTA is the GRADE EtD framework. The GRADE framework includes 7 key criteria: burden of disease, benefits and harms, values and preferences, resource use, equity, acceptability within the political system, and feasibility within the health system. An online interactive tool is available for use that allows a panel to specify their judgment, the research evidence considered, the certainty in evidence, their decision with justification, restrictions, implementation considerations, and monitoring and evaluation. GRADE EtD frameworks specify 5 options for coverage decisions: no coverage, coverage with evidence development (in the context of research), coverage with price negotiation, restricted coverage, and full coverage. Additional factors related to the health and political system have been suggested to be included in the GRADE decision-making framework to account for effective yet expensive health technologies.¹⁴

Decision criteria may improve the consistency and transparency of a decision-making process but can often be difficult to define or operationalize and may be considered too rigid.⁶⁶ However, explicit decision criteria are considered in HTA deliberative frameworks. Three aspects of the health technology under consideration include criteria for comparative and added clinical benefit, cost-effectiveness, and considerations for burden of disease or severity of disease. A select group of decision criteria are outlined in Table 2. Although the focus here is on the appraisal step of the process, there are often other processes in place at the assessment stage of the HTA review to categorize the magnitude of clinical benefit or the level of uncertainty in the evidence to help guide the appraisal step. Institut national d’excellence en santé et en services sociaux (INESSS) and HAS have set criteria to explicitly state the comparative or added clinical value of a health technology (Table 2). ICER has a process for judging the value of the health technology under consideration with respect to contextual considerations or other potential benefits or disadvantages that impact the perceived acceptable cost-effectiveness threshold. ZIN and NICE have also defined cost-effectiveness thresholds that can vary depending on other criteria or contextual factors. For example, the Netherlands has different thresholds depending on the burden of disease, and NICE considers technologies for patients with a life expectancy of less than 24 months that extend life for more than 3 months and the magnitude of the incremental therapeutic improvement as criteria for the allowance of greater thresholds (Table 2).

Navigating Uncertainty

The 2021 HTAi Global Policy Forum background document on considering and communicating uncertainty in HTA describes considerations for uncertainty in HTA according to the ITO model.⁷³ Considerations for navigating uncertainty during appraisal committee deliberations included the consistency and transparency in how uncertainties are managed between health technologies, understanding the opportunity cost of resolving uncertainty, the level of individual committee member risk tolerance, the risks associated with making a “wrong” decision or recommendation, and use of stakeholder input to resolve uncertainties. Options for appraisal committees to manage decision uncertainty included managed entry and performance-based agreements for the collection of additional evidence to reduce uncertainty. However, these options depend on the ability of the health system to enter into these arrangements.⁷⁴

Other literature considering uncertainty during appraisal committee deliberations include qualitative studies conducted by Calnan and colleagues in the context of the UK (NICE appraisal committee) and by Wranik and colleagues in the context of Poland (Polish HTA agency’s Transparency Council) and Canada (CADTH’s pan-Canadian Oncology Drug Review Expert Review Committee [pERC] and Canadian Drug Expert Committee [CDEC]). Three forms of uncertainty were identified by Calnan and colleagues: epistemic (methods used to generate knowledge), procedural, and interpersonal or relational.⁷⁴ Committee members indicated that the volume and complexity of information was a challenge, and cited concerns about access to information about the analysis and the competencies and motives of manufacturers, clinical experts, and patient members. Strategies used by the committee to address uncertainties were both informal (i.e., going with “gut feelings” or bracketing off or ignoring “potential” uncertainties and focusing on the critical ones), and more formal strategies, such as providing more technical education for committee members. Additionally, the process of discussion allowed the committee members to understand the evidence correctly, and the committee chair’s experience navigating the uncertainty and the committee members’ trust in the chair was especially important for dealing with uncertainty. Potential solutions for managing uncertainty included suggestions for instituting a decision criterion such as “intolerable uncertainty” that places the onus on the manufacturer and allows more time for committee discussions, deliberation, and preparation, and tighter controls on conflicts of interest.⁷⁴ Wranik and colleagues identified that committee members on Polish (Transparency Council) and Canadian (CADTH pERC and CDEC) committees had variable attitudes, interpretations, and tolerance for uncertainty in the clinical evidence.⁷⁵ The type of clinical uncertainty most commonly cited by committee members included issues with trial design, immature data or use of surrogate outcomes, and inappropriate comparators.⁷⁵ The authors suggested that guidance for managing uncertainty may help improve the consistency and transparency of recommendations.⁷⁵

Approaches to appropriately managing uncertainty during the deliberative appraisal process are, in part, also dictated by the level of uncertainty in the input (or assessment) phase of the HTA process. Input uncertainty is likely to continue to increase due to the emergence of technologies with less traditional forms of evidence (e.g., gene therapies, curative or life-extending therapies). Approaches for exploring and presenting uncertainty are commonplace in cost-effectiveness analyses and to some extent in comparative clinical assessments (e.g., ICERs evidence rating matrix, GRADE); however, methods to explore and manage uncertainty in other domains of HTA (e.g., stakeholder input and ethical considerations) and in balancing or considering the totality of uncertainty during deliberations are less common. The 2021 HTAi Global Policy Forum background document describes common methods to characterize uncertainty in clinical, economic, and affordability inputs of an HTA.⁷³

Openness of Deliberation

HTA agencies have different practices for stakeholder involvement in deliberations and the communication of the output of deliberations. Some agencies have fully open meetings with stakeholder attendance and involvement (e.g., ICER and ZIN), some have an open meeting for evidence presentation followed by a closed meeting for deliberation and decision-making (e.g., NICE), and others have a public record of deliberation (e.g., France provides a transcript of deliberations).³

Arriving at a Recommendation

Associated with the deliberative appraisal and decision-making processes are recommendation options. Generally, recommendation options include recommendations to reimburse, not reimburse, or reimburse with associated conditions.⁷⁶ Other recommendation options, such as those employed by NICE and the GRADE EtD framework, offer additional options that consider coverage with evidence development or coverage by a separate drug fund (Table 3). In addition to a general recommendation supporting the added value of a health technology or not supporting the added value of a health technology, recommendations such as conditions for use (e.g., specific patient population), monitoring (e.g., collection of additional data), or other guidance (e.g., implementation considerations) are often provided. In some cases, the focus of the deliberative appraisal may be to issue advice for the optimal or appropriate use of a health technology. HTA agencies have different processes to arrive at a recommendation or decision. For example, they may use voting or consensus (or a hybrid of the 2) and the voting may be open or anonymous. ICER lists the number of votes that were made in favour and not in favour of each of the key questions during the public meeting. Likewise, the number of votes for other benefits or disadvantages and contextual consideration (on a Likert scale from 1 to 3 indicating lower, intermediate, or higher long-term value for money, respectively) are also reported separately for each relevant consideration. CADTH’s Health Technology Expert Review Panel (HTERP) and NICE’s technology appraisal committee aim to reach consensus on the recommendation first (and proceed to majority voting if necessary), whereas CADTH’s CDEC and pERC committees use anonymous majority voting to reach a recommendation.

Output Step of ITO Model

Publicity of decisions is an important component of a legitimate and fair process for decision-making.⁵ This requires that recommendations and/or decisions and their rationale are fully transparent to stakeholders. Following the principle of transparency, it is recommended that the output of the deliberative appraisal include what the decision was, options or alternatives considered, facts and reasons for the decision presented, and who was involved.¹⁰ Publication of the reimbursement recommendation or decision (depending on the mandate of the HTA agency – i.e., binding or advisory) should also include the reasons for including and excluding certain criteria during the deliberations⁶ and ensure that the documentation is accessible to a broad range of stakeholders.¹⁰ In the context of jurisdictions that consider patient submissions in the review process, it was also found that patient groups are often uncertain how their submissions were weighed against other evidence.⁷⁷ It is recommended that input from stakeholders be documented and addressed explicitly and recommendations are communicated clearly and in a timely fashion.⁶ Clear communication of uncertainty in the evidence base and its impact on the committee appraisal recommendation are also critical for ensuring the consistency and transparency of the deliberative appraisal process.⁷³ Examples of ways in which HTA agencies could consider communicating uncertainty to stakeholders — patients, health technology manufacturers, and other system stakeholders — is summarized in the 2021 HTAi Global Policy Forum background document on considering and communicating uncertainty in HTA.⁷³ Examples of communication strategies may include graphical displays such as traffic light systems, lay language summaries, disclosure of meeting minutes, or dialogue between the HTA agency and the stakeholder group.⁷³

Opportunity for Feedback or Revision of the Recommendation

The opportunity to revise recommendations based on considerations from stakeholders is a key component of the accountability for reasonableness framework.⁹ In the context of drug recommendations, CADTH, NICE, SMC, and HAS issue draft (or preliminary) recommendations or guidance from appraisal committee meetings for stakeholder feedback. Not all stakeholders are eligible to provide feedback, and not all stakeholders are eligible to appeal against a recommendation. For example, NICE issues a preliminary recommendation appraisal consultation document for stakeholder feedback (sent after the appraisal took place) and allows feedback and appeals from all stakeholders (except for appeals from the public, and only under certain conditions).⁷⁸ CADTH issues draft recommendations for stakeholder feedback. Patient groups, clinician groups, manufacturers, and public payers are eligible to provide feedback on the draft recommendation. Manufacturers and public payers are eligible to file a request for reconsideration or for clarification, respectively, on the draft recommendation. Only requests for reconsideration that involve major revisions (e.g., revisions to the recommendation category or revisions to the eligible patient population) are addressed through deliberation with the full expert appraisal committee. Minor revisions (e.g., related to renewal criteria, pricing conditions) may be addressed by a subpanel of the expert appraisal committee. ICER, PBAC, and INESSS do not issue a preliminary recommendation document for stakeholder feedback.

Additional Considerations

The deliberative appraisal process is subject to changes that occur across all aspects of HTA and the health system (e.g., changes in regulatory practices, innovations in health technologies, the COVID-19 pandemic). Select additional and emerging considerations for the deliberative process are discussed here, including the life-cycle approach to HTA, real-world evidence, responding to the need for virtual meetings, efficiency and sustainability of appraisal committees, and fluidity of the deliberative appraisal process.

Life-Cycle Approach to HTA

It is increasingly recognized that HTA should occur across the life cycle of a technology.¹ Traditionally, deliberative appraisal processes have been developed and implemented to respond to implementation questions for new and/or emerging technologies; however, the need for disinvestment questions is also apparent. In the existing literature there is a lack of a consistent terminology to describe disinvestment (also termed resource withdrawal).⁷⁹ This emphasizes the importance of — and challenges to — explicit disinvestment decisions becoming commonplace.⁸⁰ Public perspectives on disinvestments of cancer drugs were elicited from a public deliberation event in Canada, which provided some insight into the public’s views.⁵⁴ The Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) collaboration has suggested that the current deliberative framework for reviewing oncology drugs at a national level in Canada could be used (with modification) to accommodate a reassessment process. For example, changing the current Adoption Feasibility criteria of the framework to Implementation and Sustainability which could consider factors such as “drug utilization, budget impact analysis, and an assessment of if and how the funding algorithm has changed since the initial review” was suggested as a modification to the deliberative framework that would allow for its use for questions of reassessment.⁸¹ Furthermore, the collaborative proposed 3 separate recommendation categories in response to a reassessment review: status quo, revisit funding criteria or pricing, do not continue funding, and/or delist.

Real-World Evidence

The use of real-world data (RWD) and real-world evidence (RWE) in HTA is a rapidly evolving area with implications for the deliberative appraisal process. Several definitions for RWD and RWE are used, and there is a lack of consensus regarding how RWD and RWE should be defined, how it is generated, and what it may inform.⁸² RWD is most commonly defined as data collected outside of an RCT; however, others define RWD as data collected in a non-interventional and/or non-controlled setting or data collected in a non-experimental setting.⁸² There is also recognition that a consensus definition would be most valuable if applied consistently across the product life cycle (i.e., regulatory, HTA, and post-market). Several collaborative initiatives are under way (or have been completed) to define and address the appropriate use of RWD in decision-making, including a working group convened by the Council for International Organizations of Medical Sciences, the EU-based GetReal Initiative, the Canadian Real-World Evidence Action Plan,⁸³ and CanREValue.⁸¹ RWD can have different roles across the HTA process: within the assessment or input phase (e.g., clinical and economic evidence reports), during the appraisal or throughput stage (e.g., from stakeholder input and the experiences and perspectives of committee members during committee meetings), and in the output stage (e.g., in the form of stakeholder feedback) or implementation and monitoring of the decision. Inclusion of RWD during the assessment phase will have implications for committee deliberations because RWD is often associated with greater uncertainty in the level of evidence and is often generated to fill gaps and unknowns or to provide context to traditional forms of evidence (e.g., data for underrepresented patient populations or for outcomes that are not available from conventional clinical trials).⁸⁴ The collection of RWD may be used to inform conditional reimbursement by continuing to collect data on uncertain outcomes.⁸⁴ As the use of RWD in HTA evolves, deliberative appraisal processes will need to be established that appropriately acknowledge and manage RWE in complement with more traditional forms of evidence. Appropriate ways to communicate the use of RWE in the output of the deliberation (i.e., the recommendation document or associated documents) will also be needed.

Responding to the Need for Virtual Meetings

In-person or hybrid in-person and virtual committee meetings were commonplace for some organizations before the COVID-19 pandemic. However, in response to the pandemic, many HTA agency committees have initiated fully virtual meetings to maintain operations. Early commentary from HTA, regulatory, and even parliamentary committee meetings has acknowledged the potential challenges and opportunities of the virtual environment. For example, a report from the Standing Committee on Procedure and House Affairs in Canada identified challenges of voting during the COVID-19 pandemic.⁸⁵ These included confidentially and cybersecurity concerns, reduced personal interaction between committee members, and issues with technical connectivity, sound, and video capabilities, which may also impact inclusion of those individuals in rural or remote locations or with limited computer skills.⁸⁵ Potential opportunities in the virtual environment included the potential to better accommodate individuals for whom travel or family obligations may have previously precluded participation and the potential to give everyone a “front row seat” compared with in-person meetings. There is also a reduction in travel costs.⁸⁶ Identified strategies for working in a virtual environment included requiring video be turned on, ensuring adequate lighting, and establishing means of speaking (e.g., raise hand, chat box).⁸⁵

Efficiency and Sustainability of Appraisal Committees

The growth in the number and complexity of new, emerging health technologies necessitates the need for an efficient and a sustainable committee deliberative appraisal process. Multiple committee meetings for a single technology review are common at NICE; only 20% of technologies reach a final decision after 1 meeting, and 73% of initially negative recommendation become positive in the final recommendation.⁸⁷ NICE has introduced reforms to address operational efficiencies. For example, a fast-track appraisal process was introduced in 2018 for technologies that offer obvious and significant benefit over existing practice with ICERs less than £10,000 per QALY. Additionally, when cost is the only concern after an initial recommendation, and the manufacturer brings forward a patient access scheme discount, the chair may make a final recommendation without going back for another committee meeting.⁸⁷ Recent recommendations in France (HAS) to discontinue automatic reassessment of technologies every 5 years if there are no “issues in terms of efficacy, safety, or role in the therapeutic strategy” also speak to the need for addressing process efficiencies.⁸⁸ A 2018 report visioned Canada’s health and biosciences sector doubling the size of its health and biosciences sector by 2025.⁸⁹ The report identified the need for an “agile and streamlined regulatory and procurement approaches to support access to value-based innovations”⁸⁹ — a process that HTA appraisal committees play a key part in.

Fluidity of the Deliberative Appraisal Processes

A deliberative appraisal process with well-defined criteria that are in line with and rooted in health system values is needed. System values may change over time; therefore, it is important to have a deliberative appraisal process that is set up to respond to these changes. For example, the evidence-informed deliberative process framework recommends that defining decision criteria be reassessed every 3 years to 5 years.⁶

Assessing the Quality of the Deliberative Appraisal Process in HTA

No “gold standard” exists to assess the quality of deliberative appraisal processes,⁹⁰ and features of deliberative processes in HTA are expected to vary according to the jurisdiction and decision context under consideration.^7,10,16 Although there is limited empirical literature assessing specific elements of deliberative appraisal processes in HTA, it is important to assess the quality of the process and understand the impact on decision-making, especially when complex and controversial decision-making is required.^7,10,16 Frameworks and principles have emerged to support the development of jurisdiction and context-specific deliberative processes that are expected to lead to reimbursement decisions that are acceptable to all stakeholders. Table 4 outlines a sample of criteria, principles, and frameworks that range from overarching concepts in ethics (i.e., accountability for reasonableness) and HTA (i.e., methods, criteria for assessing value, and taking a life-cycle approach) to discipline-specific principles and evaluative criteria (e.g., patient and public involvement and rare disease drug considerations) that can be used as a starting place to evaluate the quality of deliberative appraisal processes.

Limitations

Although this literature search covered multiple relevant peer-reviewed databases, grey literature, and corresponding reference lists, the date range was limited to the previous 5 years and the search was restricted to literature related to HTA. Consequently, key literature may have been missed, including literature from other disciplines that could help inform deliberative appraisal processes in HTA. The methods used also did not follow systematic review methodology, such as including duplicate screening or an assessment of the quality of the included studies. However, it is expected that any key literature that may have been missed in this search or through single reviewer screening would have been identified through a review of the reference lists of key publications and in conversation with internal and external content experts. Details of a limited selection of national and international HTA agency deliberative appraisal processes have been included, so it is possible that relevant practices in other countries were missed. The purpose of this review was to assess practices that may be relevant for the Canadian context; therefore, the reference countries selected were thought to best reflect what might be most inspirational for Canada. Finally, this interpretation of deliberative appraisal processes is influenced by the authors’ own conceptualizations of the deliberative process and the aspects deemed most relevant to discuss. The aim was to be as comprehensive as possible, be explicit about the methods used, and highlight areas of uncertainty or where limited evidence exists.

Strengths

This review focused on literature relevant for the Canadian context and examined considerations for all technologies, not just pharmaceuticals, which appeared to be the predominant focus in the literature. The existing evidence base for deliberative appraisal processes in HTA was conceptualized, allowing readers to understand the relationships between existing frameworks in a way that has not been described previously. Finally, items less commonly discussed within the context of the deliberative appraisal process in HTA were highlighted, such as taking a life-cycle approach to HTA, responding to virtual meetings, considerations for real-world evidence, efficiency and sustainability of appraisal committees, and the fluidity of the deliberative appraisal process.

Conclusions and Implications for Decision- or Policy-Making

The use of deliberative appraisal processes is a critical step in the HTA process to ensure that consistent, credible, and transparent decision-making is achieved. There is wide variation in the practices used nationally and internationally, such as the extent of stakeholder involvement, the structure and size of committees, the role of the chair, the openness of meetings, and the extent to which decision rules are applied. Deliberative appraisal processes require continual monitoring and evaluation to facilitate a consistent, credible, and transparent process, and to ensure appropriate alignment with societal values and the needs of decision-makers. Deliberative processes are a complex field of inquiry with limited consensus on definitions, components, and best practices. Most of the literature identified focused on pharmaceutical products; however, it is often reasonable to extrapolate to non-pharmaceutical products (keeping contextual elements in mind). No gold standard deliberative appraisal process exists and is likely to vary according to jurisdictional context; however, principles, frameworks, and action items have emerged to guide the appropriate design of processes for HTA. Existing literature recognizes the need for more empirical literature to assess the impact of various components of the deliberative process on decision outcomes. This review presented a conceptualization of what is meant by deliberative processes in HTA. Future work and research should target the development of a consensus definition for deliberative processes in HTA which will allow for design and element consistency and incorporation of emerging evidence on deliberative processes.

References

1.O'Rourke B, Oortwijn W, Schuller T. The new definition of health technology assessment: a milestone in international collaboration. Int J Technol Assess Health Care. 2020;36(3):187-190. PubMed

2.Kristensen FB, Husereau D, Huic M, Drummond M, Berger ML, Bond K, et al. Identifying the need for good practices in health technology assessment: summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. Value Health. 2019;22(1):13-20. PubMed

3.Bond K. Deliberative processes in health technology assessment: prospects, problems, and policy proposals. HTAi Global Policy Forum Background Paper. 2020; https://htai.org/wp-content/uploads/2020/02/HTAi_GPF-newOrleans_program_background-paper.pdf. Accessed 2021 Feb 9.

4.Hofmann B, Cleemput I, Bond K, Krones T, Droste S, Sacchini D, et al. Revealing and acknowledging value judgments in health technology assessment. Int J Technol Assess Health Care. 2014;30(6):579-586. PubMed

5.Daniels N, Porteny T, Urritia J. Expanded HTA: enhancing fairness and legitimacy. Int J Health Policy Manag. 2015;5(1):1-3. PubMed

6.Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes. A practical guide for HTA agencies to enhance legitimate decision-making. Version 1.0. Nijmegen (Netherlands): Radboud University Medical Centre, Radboud Institute for Health Sciences.; 2019: https://www.radboudumc.nl/getmedia/17a96fdb-553b-4e68-81ab-4d8d9a7f9ff1/UMCRadboud_Guide_17x24_inside_DEF_WEB.aspx. Accessed 2021 Feb 9.

7.Oortwijn W, Jansen M, Baltussen R. Use of evidence-informed deliberative processes by health technology assessment agencies around the globe. Int J Health Policy Manag. 2020;9(1):27-33. PubMed

8.Salas-Vega S, Bertling A, Mossialos E. A comparative study of drug listing recommendations and the decision-making process in Australia, the Netherlands, Sweden, and the UK. Health Policy. 2016;120(10):1104-1114. PubMed

9.Daniels N, van der Wilt GJ. Health technology assessment, deliberative process, and ethically contested issues. Int J Technol Assess Health Care. 2016;32(1-2):10-15. PubMed

10.Bond K, Stiffell R, Ollendorf DA. Principles for deliberative processes in health technology assessment. Int J Technol Assess Health Care. 2020 Aug 4:1-8. PubMed

11.Kristensen FB, Lampe K, Wild C, Cerbo M, Goettsch W, Becla L. The HTA Core Model® - 10 years of developing an international framework to share multidimensional value assessment. Value Health. 2017;20(2):244-250. PubMed

12.Wahlster P, Brereton L, Burns J, Hofmann B, Mozygemba K, Oortwijn W, et al. An integrated perspective on the assessment of technologies: INTEGRATE-HTA. Int J Technol Assess Health Care. 2017;33(5):544-551. PubMed

13.Tarricone R, Torbica A, Drummond M, MedtecHTA Project Group. Key recommendations from the MedtecHTA project. Health Econ. 2017;26 Suppl 1:145-152. PubMed

14.Morgan RL, Kelley L, Guyatt GH, Johnson A, Lavis JN. Decision-making frameworks and considerations for informing coverage decisions for healthcare interventions: a critical interpretive synthesis. J Clin Epidemiol. 2018;94:143-150. PubMed

15.Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. PubMed

16.Baltussen R. Multicriteria decision analysis to support health technology assessment agencies: benefits, limitations, and the way forward. Value Health. 2019;22(11):1283-1288. PubMed

17.Baltussen R, Jansen M, Bijlmakers L. Stakeholder participation on the path to universal health coverage: the use of evidence-informed deliberative processes. Trop Med Int Health. 2018;23(10):1071-1074. PubMed

18.Oortwijn W, Sampietro-Colom L, Habens F. Developments in value frameworks to inform the allocation of healthcare resources. Int J Technol Assess Health Care. 2017;33(2):323-329. PubMed

19.Institute for Clinical and Economic Review. 2020-2023 value assessment framework. [Boston (MA)]: ICER; 2020 (updated 2020 Oct 23): https://icer.org/wp-content/uploads/2020/10/ICER_2020_2023_VAF_102220.pdf. Accessed 2021 Feb 9.

20.Clark S, Weale A. Social values in health priority setting: a conceptual framework. J Health Organ Manag. 2012;26(3):293-316. PubMed

21.Whitty JA, Littlejohns P. Social values and health priority setting in Australia: an analysis applied to the context of health technology assessment. Health Policy. 2015;119(2):127-136. PubMed

22.National Institute for Health and Care Excellence. The principles that guide the development of NICE guidance and standards. 2020; https://www.nice.org.uk/about/who-we-are/our-principles#introduction. Accessed 2020 May 28.

23.Bombard Y, Abelson J, Simeonov D, Gauvin FP. Eliciting ethical and social values in health technology assessment: a participatory approach. Soc Sci Med. 2011;73(1):135-144. PubMed

24.O’Keefe-Markman C, Lea KD, McCabe C, Hyshka E, Bubela T. Social values for health technology assessment in Canada: a scoping review of hepatitis C screening, diagnosis and treatment. BMC Public Health. 2020;20(1):89. PubMed

25.Rosina J, Rogalewicz V, Ivlev I, Jurickova I, Donin G, Jantosova N, et al. Health technology assessment for medical devices. Lekar a Technika. 2014;44(3):23-36.

26.Drummond M, Griffin A, Tarricone R. Economic evaluation for devices and drugs--same or different? Value Health. 2009;12(4):402-404. PubMed

27.Enzing JJ, Knies S, Boer B, Brouwer WBF. Broadening the application of health technology assessment in the Netherlands: a worthwhile destination but not an easy ride? Health Econ Policy Law. 2020 Aug 6:1-17. PubMed

28.Nicod E, Whittal A, Drummond M, Facey K. Are supplemental appraisal/reimbursement processes needed for rare disease treatments? An international comparison of country approaches. Orphanet J Rare Dis. 2020;15(1):189. PubMed

29.Annemans L, Ayme S, Le Cam Y, Facey K, Gunther P, Nicod E, et al. Recommendations from the European Working Group for Value Assessment and Funding Processes in Rare Diseases (ORPH-VAL). Orphanet J Rare Dis. 2017;12(1):50. PubMed

30.Facey K, Granados A, Guyatt G, Kent A, Shah N, Van Der Wilt GJ, et al. Generating health technology assessment evidence for rare diseases. Int J Technol Assess Health Care. 2014;30(4):416-422. PubMed

31.Nicod E, Annemans L, Bucsics A, Lee A, Upadhyaya S, Facey K. HTA programme response to the challenges of dealing with orphan medicinal products: process evaluation in selected European countries. Health Policy. 2019;123(2):140-151. PubMed

32.Hettle R, Corbett M, Hinde S, Hodgson R, Jones-Diette J, Woolacott N, et al. The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal. Health Technol Assess. 2017;21(7):1-204. PubMed

33.Short H, Stafinski T, Menon D. A national approach to reimbursement decision-making on drugs for rare diseases in Canada? Insights from across the ponds. Healthc Policy. 2015;10(4):24-46. PubMed

34.Schey C, Krabbe PF, Postma MJ, Connolly MP. Multi-criteria decision analysis (MCDA): testing a proposed MCDA framework for orphan drugs. Orphanet J Rare Dis. 2017;12(1):10. PubMed

35.Winquist E, Coyle D, Clarke JTR, Evans GA, Seager C, Chan W, et al. Application of a policy framework for the public funding of drugs for rare diseases. J Gen Intern Med. 2014;29(Suppl 3):S774-S779. PubMed

36.Adkins EM, Nicholson L, Floyd D, Ratcliffe M, Chevrou-Severac H. Oncology drugs for orphan indications: how are HTA processes evolving for this specific drug category? ClinicoEcon. 2017;9:327-342. PubMed

37.Dixon P, Chamberlain C, Hollingworth W. Did it matter that the Cancer Drugs Fund was not NICE? A retrospective review. Value Health. 2016;19(6):879-884. PubMed

38.Kleinhout-Vliek T, de Bont A, Boysen M, Perleth M, van der Veen R, Zwaap J, et al. Around the tables – contextual factors in healthcare coverage decisions across Western Europe. Int J Health Policy Manag. 2020;9(9):390-402. PubMed

39.EUnetHTA Joint Action 2, Work Package 8. HTA Core Model® version 3.0. 2016; www.htacoremodel.info/BrowseModel.aspx. . Accessed 2021 Feb 9.

40.Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012;10(1):9. PubMed

41.Hayati R, Bastani P, Kabir MJ, Kavosi Z, Sobhani G. Scoping literature review on the basic health benefit package and its determinant criteria. Global Health. 2018;14(1):26. PubMed

42.Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. PubMed

43.Ollendorf D, Krubiner C. The dynamics of health technology assessment: is it just about the evidence? Washington (DC): Center for Global Development; 2019: https://www.cgdev.org/blog/dynamics-health-technology-assessment-it-just-about-evidence. Accessed 2021 Feb 9.

44.Jansen MPM, Baltussen R, Bærøe K. Stakeholder participation for legitimate priority setting: a checklist. Int J Health Policy Manag. 2018;7(11):973-976. PubMed

45.Health Canada. Health Canada policy toolkit for public involvement in decision making. Ottawa: Health Canada; 2000: https://www.canada.ca/content/dam/hc-sc/migration/hc-sc/ahc-asc/alt_formats/pacrb-dgapcr/pdf/public-consult/2000decision-eng.pdf. Accessed 2021 Feb 9.

46.Facey K, Henshall C, Sampietro-Colom L, Thomas S. Improving the effectiveness and efficiency of evidence production for health technology assessment. Int J Technol Assess Health Care. 2015;31(4):201-206. PubMed

47.Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985-991. PubMed

48.Abelson J, Wagner F, DeJean D, Boesveld S, Gauvin FP, Bean S, et al. Public and patient involvement in health technology assessment: a framework for action. Int J Technol Assess Health Care. 2016;32(4):256-264. PubMed

49.Street J, Stafinski T, Lopes E, Menon D. Defining the role of the public in health technology assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. 2020;36(2):87-95. PubMed

50.Chim L, Salkeld G, Kelly PJ, Lipworth W, Hughes DA, Stockler MR. Community views on factors affecting medicines resource allocation: cross-sectional survey of 3080 adults in Australia. Aust Health Rev. 2019;43(3):254-260. PubMed

51.Wortley S, Tong A, Howard K. Community views and perspectives on public engagement in health technology assessment decision making. Aust Health Rev. 2017;41(1):68-74. PubMed

52.Baran-Kooiker A, Atikeler K, Gaitova K, Holownia-Voloskova M, Turcu-Stiolica A, Kooiker C, et al. Applicability of the EVIDEM multi-criteria decision analysis framework for orphan drugs - results from a study in 7 Eurasian countries. Acta Pol Pharm. 2019;76(3):581-598.

53.Johnson AP, Sikich NJ, Evans G, Evans W, Giacomini M, Glendining M, et al. Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25(2):141-150. PubMed

54.Costa S, Bentley C, Regier DA, McTaggart-Cowan H, Mitton C, Burgess MM, et al. Public perspectives on disinvestments in drug funding: results from a Canadian deliberative public engagement event on cancer drugs. BMC Public Health. 2019;19(1):977. PubMed

55.Abelson J. Patient engagement in health technology assessment: what constitutes ‘meaningful’ and how we might get there. J Health Serv Res Policy. 2018;23(2):69-71. PubMed

56.Douglas H. Weighing complex evidence in a democratic society. Kennedy Inst Ethics J. 2012;22(2):139-162. PubMed

57.Morestin F, Gauvin F-P, Hogue M-C, Benoit F. Method for synthesizing knowledge about public policies. Montréal: National Collaborating Centre for Healthy Public Policy, Institut national de santé publique du Québec; 2010: http://www.ncchpp.ca/docs/methodpp_en.pdf. Accessed 2021 Feb 9.

58.Haute Autorité de santé. Commission nationale d’évaluation des dispositifs médicaux et des technologies de santé. 2019; https://www.has-sante.fr/jcms/c_419486/fr/commission-nationale-d-evaluation-des-dispositifs-medicaux-et-des-technologies-de-sante. Accessed 2020 Dec 5.

59.Barnieh L, Manns B, Harris A, Blom M, Donaldson C, Klarenbach S, et al. A synthesis of drug reimbursement decision-making processes in organisation for economic co-operation and development countries. Value Health. 2014;17(1):98-108. PubMed

60.Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1-13. PubMed

61.Marsh K, Ijzerman M, Thokala P, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125-137. PubMed

62.Garau M, Devlin NJ. Using MCDA as a decision aid in health technology appraisal for coverage decisions: opportunities, challenges and unresolved questions. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, eds. Multi-criteria decision analysis to support healthcare decisions. Cham (Switzerland): Springer International Publishing; 2017:277-298.

63.Simoens S. Health technologies for rare diseases: does conventional HTA still apply? Expert Rev Pharmacoecon Outcomes Res. 2014;14(3):315-317. PubMed

64.van Til JA, Groothuis-Oudshoorn CG, Baltussen R, Zwaap J, Boer B, Ijzerman MJ. The value of MCDA in health technology assessment: a explorative study in the Dutch context. Value Health. 2016;19(7):A469-A470.

65.DiStefano MJ, Krubiner CB. Beyond the numbers: a critique of quantitative multi-criteria decision analysis. Int J Technol Assess Health Care. 2020;36(4):292-296. PubMed

66.Parmelli E, Amato L, Oxman AD, Alonso-Coello P, Brunetti M, Moberg J, et al. GRADE Evidence to Decision (EtD) framework for coverage decisions. Int J Technol Assess Health Care. 2017;33(2):176-182. PubMed

67.Institut national d’excellence en santé et en services sociaux. Evaluation of drugs for listing purposes: a change of approach. Québec (QC): INESSS; 2018: https://www.inesss.qc.ca/fileadmin/doc/INESSS/Inscription_medicaments/Processus/Evaluation_of_drugs_a_change_of_approach.pdf. Accessed 2021 Feb 11.

68.Haute Autorité de santé. Methods and criteria for assessing medical devices. 2015; https://www.has-sante.fr/jcms/c_2035654/en/methods-and-criteria-for-assessing-medical-devices. Accessed 2021 Feb 11.

69.National Institute for Health and Care Excellence. Guide to the methods of technology appraisal 2013. Chapter 6: The appraisal of the evidence and structured decision-making. 2013; https://www.nice.org.uk/process/pmg9/chapter/the-appraisal-of-the-evidence-and-structured-decision-making. Accessed 2021 Feb 11.

70.National Institute for Health and Care Excellence. Changes to NICE drug appraisals: what you need to know. 2017; https://www.nice.org.uk/news/feature/changes-to-nice-drug-appraisals-what-you-need-to-know. Accessed 2021 Feb 11.

71.Zwaap J, Knies S, van der Meijden C, Staal P, van der Heiden L. Cost-effectiveness in practice. [Diemen (Netherlands)]: Zorginstituut Nederland; 2015: https://english.zorginstituutnederland.nl/binaries/zinl-eng/documents/reports/2015/06/16/cost-effectiveness-in-practice/Cost-effectiveness+in+practice.pdf. Accessed 2021 Feb 11.

72.Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. General methods. Version 5.0 of 10 July 2017. Köln (Germany): IQWiG; 2017: https://www.iqwig.de/methoden/general-methods_version-5-0.pdf?rev=117386. Accessed 2021 Feb 11.

73.Health Technology Assessment International. Considering and communicating uncertainty in HTA [draft]. (HTAi Global Policy Forum 2021 background paper). Edmonton (AB): HTAi; 2020: https://htai.org/wp-content/uploads/2020/12/2021-GPF-Background-Paper_Draft.pdf. Accessed 2021 Feb 9.

74.Calnan M, Hashem F, Brown P. Still elegantly muddling through? NICE and uncertainty in decision making about the rationing of expensive medicines in England. Int J Health Serv. 2017;47(3):571-594. PubMed

75.Wranik WD, Gambold L, Peacock S. Uncertainty tolerance among experts involved in drug reimbursement recommendations: qualitative evidence from HTA committees in Canada and Poland. Health Policy. 2020 Dec 24. PubMed

76.Allen N, Liberti L, Walker SR, Salek S. A comparison of reimbursement recommendations by European HTA agencies: is there opportunity for further alignment? Front Pharmacol. 2017;8:384. PubMed

77.Scott AM, Wale JL, HTAi Patient and Citizen Involvement in HTA Interest Group, Patient Involvement and Education Working Group. Patient advocate perspectives on involvement in HTA: an international snapshot. Res Involv Engagem. 2017;3(1):2. PubMed

78.Cowling T, Pant S. Single drug technology assessment processes across health technology assessment organizations. (CADTH Environmental scan no. 55). Ottawa: CADTH; 2016: https://www.cadth.ca/sites/default/files/pdf/ES0293_Single_Technology_Assessment_Processes.pdf. Accessed 2021 Feb 9.

79.Embrett M, Randall GE, Lavis JN, Dion ML. Conceptualising characteristics of resources withdrawal from medical services: a systematic qualitative synthesis. Health Res Policy Syst. 2020;18(1):123-123. PubMed

80.Calabrò GE, La Torre G, de Waure C, Villari P, Federici A, Ricciardi W, et al. Disinvestment in healthcare: an overview of HTA agencies and organizations activities at European level. BMC Health Serv Res. 2018;18(1):148. PubMed

81.Chan K, Nam S, Evans B, de Oliveira C, Chambers A, Gavura S, et al. Developing a framework to incorporate real-world evidence in cancer drug funding decisions: the Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration. BMJ Open. 2020;10(1):e032884. PubMed

82.Makady A, de Boer A, Hillege H, Klungel O, Goettsch W. What is real-world data? A review of definitions based on literature and stakeholder interviews. Value Health. 2017;20(7):858-865. PubMed

83.Tadrous M, Ahuja T, Ghosh B, Kropp R. Developing a Canadian real-world evidence action plan across the drug life cycle. Healthc Policy. 2020;15(4):41-47. PubMed

84.Canadian Agency for Drugs and Technologies in Health, Canadian Association for Population Therapeutics, Health Canada, Institute of Health Economics. Defining decision-grade real-world evidence and its role in the Canadian context: a design sprint. Summary report of a workshop: October 21, 2018. Edmonton (AB): Institute of Health Economics; 2018: https://www.ihe.ca/advanced-search/defining-decision-grade-real-world-evidence-and-its-role-in-the-canadian-context-a-design-sprint-summary-report-of-a-workshop. Accessed 2021 Feb 9.

85.Carrying out members’ parliamentary duties: the challenges of voting during the COVID-19 pandemic. Report of the Standing Committee on Procedure and House Affairs. Ottawa: House of Commons of Canada; 2020: https://www.ourcommons.ca/Content/Committee/431/PROC/Reports/RP10833363/procrp07/procrp07-e.pdf. Accessed 2021 Feb 9.

86.National Institute for Health and Care Excellence. National Institute for Health and Care Excellence public board meeting – meeting as the board committee: Wednesday 20 May 2020 at 1:30pm via Zoom. 2020; https://www.nice.org.uk/Media/Default/Get-involved/Meetings-In-Public/Public-board-meetings/agenda-and-papers-may-20.pdf. Accessed 2021 Feb 9.

87.Walton MJ, O'Connor J, Carroll C, Claxton L, Hodgson R. A review of issues affecting the efficiency of decision making in the NICE Single Technology Appraisal process. Pharmacoeconom Open. 2019;3(3):403-410. PubMed

88.Haute Autorité de santé. Innovative medicines assessment action plan. Saint-Denis (France): HAS; 2020: https://www.has-sante.fr/upload/docs/application/pdf/2020-03/innovative_medicine_action_plan_27.01.20.pdf. Accessed 2021 Feb 9.

89.Report from Canada's economic strategy tables: health and biosciences. Ottawa: Government of Canada; 2018: https://www.ic.gc.ca/eic/site/098.nsf/eng/00025.html. Accessed 2021 Feb 9.

90.De Vries R, Stanczyk A, Wall IF, Uhlmann R, Damschroder LJ, Kim SY. Assessing the quality of democratic deliberation: a case study of public deliberation on the ethics of surrogate consent for research. Soc Sci Med. 2010;70(12):1896-1903. PubMed

91.De Vries R, Stanczyk AE, Ryan KA, Kim SYH. A framework for assessing the quality of democratic deliberation: enhancing deliberation as a tool for bioethics. J Empir Res Hum Res Ethics. 2011;6(3):3-17. PubMed

92.Donelan R, Walker S, Salek S. The development and validation of a generic instrument, QoDoS, for assessing the quality of decision making. Front Pharmacol. 2016;7:180. PubMed

93.Rosenberg-Yunger ZRS, Bayoumi AM. Evaluation criteria of patient and public involvement in resource allocation decisions: a literature review and qualitative study. Int J Technol Assess Health Care. 2017;33(2):270-278. PubMed

94.Rowe G, Frewer L. Public participation methods: a framework for evaluation. Sci Technol Human Values. 2000;25(1):3-29.

95.Facey K, Boivin A, Gracia J, Hansen HP, Lo Scalzo A, Mossman J, et al. Patients' perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334-340. PubMed

96.Mason RJ, Searle KM, Bombard Y, Rahmadian A, Chambers A, Mai H, et al. Evaluation of the impact of patient involvement in health technology assessments: a scoping review. Int J Technol Assess Health Care. 2020;36(3):217-223. PubMed

97.Staley K, Doherty C. It's not evidence, it's insight: bringing patients' perspectives into health technology appraisal at NICE. Res Involv Engagem. 2016;2:4. PubMed

98.Bouvy JC, Cowie L, Lovett R, Morrison D, Livingstone H, Crabb N. Use of patient preference studies in HTA decision making: a NICE perspective. Patient. 2020;13(2):145-149. PubMed

99.CADTH. Guidelines for the economic evaluation of health technologies: Canada. 4th ed. Ottawa: CADTH; 2017: https://www.cadth.ca/dv/guidelines-economic-evaluation-health-technologies-canada-4th-edition. Accessed 2021 Feb 9.

100.Crispi F, Naci H, Barkauskaite E, Osipenko L, Mossialos E. Assessment of devices, diagnostics and digital technologies: a review of NICE medical technologies guidance. Appl Health Econ Health Policy. 2019;17(2):189-211. PubMed

101.National Institute for Health and Care Excellence. Guide to the processes of technology appraisal. 2014 (updated 2018); https://www.nice.org.uk/process/pmg19/. Accessed 2021 Feb 9.

102.Thokala P, Dixon S, Jahn B. Resource modelling: the missing piece of the HTA jigsaw? PharmacoEcon. 2015;33(3):193-203. PubMed

103.Stafinski T, Deber R, Rhainds M, Martin J, Noseworthy T, Bryan S, et al. The introduction of new non-drug health technologies into Canadian healthcare institutions: opportunities and challenges. Healthc Policy. 2019;15(1):95-106. PubMed

104.Widrig D, Tag B. HTA and its legal issues: a framework for identifying legal issues in health technology assessment. Int J Technol Assess Health Care. 2014;30(6):587-594. PubMed

105.Flood CM, Chen YY. Charter rights & health care funding: a typology of Canadian health rights litigation. Ann Health Law. 2010;19(3):479-526. PubMed

106.Abrishami P, Oortwijn W, Hofmann B. Ethics in HTA: examining the “need for expansion”. Int J Health Policy Manag. 2017;6(10):551-553. PubMed

107.Lewis JR, Kerridge I, Lipworth W. Coverage with evidence development and managed entry in the funding of personalized medicine: practical and ethical challenges for oncology. J Clin Oncol. 2015;33(34):4112-4117. PubMed

108.Polisena J, De Angelis G, Kaunelis D, Gutierrez-Ibarluzea I. Environmental impact assessment of a health technology: a scoping review. Int J Technol Assess Health Care. 2018;34(3):317-326. PubMed

109.Marsh K, Ganz ML, Hsu J, Strandberg-Larsen M, Gonzalez RP, Lund N. Expanding health technology assessments to include effects on the environment. Value Health. 2016;19(2):249-254. PubMed

Appendix 1: Criteria Considered in Select Assessment and Decision Frameworks

Table 5: Criteria Considered in Select Examples of Assessment and Decision Frameworks

EtD = evidence to decision; EUnetHTA = European Network for Health Technology Assessment; INTEGRATE-HTA = Integrated HTA for evaluating complex technologies.

Table 6: Domains and Subdomains of Criteria for Consideration in HTA Assessment and Appraisal Processes

The criteria captured within the classification system used by Guindo et al. and Havati et al. have been listed as example subdomains within the 9 general domains identified in this review.

Sources: Guindo et al.⁴⁰ and Hayati et al.⁴¹

Appendix 2: Summary of Common Evidentiary Domains Informing the Deliberative Appraisal

Patient and Caregiver Perspectives

Patient perspectives include information gathered from a patient’s perspective related to health, disease, or a particular health technology, such as a patients’ experiences, attitudes, beliefs, values, and expectations.⁹⁵ The type of patient involvement may include consultation with the patient (or caregiver), participation in HTA appraisal committees, or through patient experience submissions.⁹⁶ The value of patients’ perspectives is widely acknowledged in HTA. The mechanisms by which patients’ perspectives add value to committee deliberations may include understanding the real-life nature of the condition and the clinical benefits and side effects, understanding the elements of the health technology that matter most to patients and caregivers, and providing committees with insight into the relevance of the clinical and economic research to patients.⁹⁷ Findings from Staley and Doherty⁹⁷ suggested that patient perspectives should not necessarily be weighed against the clinical and economic evidence but should be used to consider the clinical and economic data in a different light. Additional considerations related to patient and carer perspectives are outlined in the Throughput section of Stakeholder Involvement.

Patients’ perspectives are distinguished here from patients’ preferences. Patients’ preferences are quantitative measurements of their preferences. Patient preference studies may be used to inform reimbursement recommendations and decision-making by HTA agencies. They can be used to understand patient insights beyond that those represented in input during the appraisal process or during committee meetings, to understand how people with a particular condition make trade-offs between technologies available to them, or to identify benefits of a health technology when they are not well-captured in the clinical or economic evidence.⁹⁸ Patient preference studies are not routinely used in HTA; however, opportunities to use this type of evidence have been identified.⁹⁸

Clinical Evidence

Clinical evidence considerations typically refer to the efficacy, safety, and effectiveness of a health technology and interpret the relative efficacy and safety of a health technology compared to standard practice. Clinical evidence considerations during deliberations are typically informed by a systematic review and an appraisal of the literature. This may include direct and indirect evidence, although HTA agencies vary in the types of evidence considered in their reviews (e.g., RCTs, observational studies, and RWE). The assessment is typically done by the HTA agency itself or outsourced to an academic unit and then appraised during the deliberative process by the expert or appraisal committee. Several HTA agencies have implemented guidance for including a consistent measure of the actual clinical benefit and the relative clinical benefit, including ICER, HAS, and INESSS (incremental or similar). The committee then appraises this evidence alongside other criteria in the framework. There is considerable overlap between the clinical and economic domains given that clinical evidence is often used to inform the economic analyses.

The specific subdomains considered within the clinical criteria vary between jurisdictions, but may include considerations of efficacy, safety, effectiveness, effect on symptoms, progression, ability to work, activities of daily living, need for hospitalization, patient-reported outcomes, quality of care, impact on non–health-related quality of life, patient satisfaction, and consideration of the strength, certainty, and relevance of the evidence available. The evidence “bar” and weight of each of these individual components of evidence, how these may change depending on the contextual considerations (e.g., type of medical condition or health technology), and how these considerations are discussed during the deliberative process are not well-understood.

Economic Evidence

Economic evidence refers to analyses that assess the trade-offs between the costs and effects of 2 or more health technologies (or doing nothing or using standard of care).⁹⁹ The extent to which economic evidence is used in the deliberative framework and process varies across jurisdictions. The use of economic evidence may range from being a key driver of a reimbursement recommendation to not being a mandated component of the HTA and deliberative process. IQWIG and HAS (Transparency Committee), for example, do not mandate a consideration of economic evidence, whereas NICE has identified cost-effectiveness as a main driver (along with clinical evidence) for recommendations.¹⁰⁰ Although economic evidence is consistently a key criterion identified explicitly by HTA agencies,⁴² it may not always be a critical component of a list or do not list recommendation. In Canada, INESSS will only review the cost-effectiveness evidence if the therapeutic value of the health technology has been established. Without a mandated upper incremental cost-effectiveness threshold for technologies in Canada (CADTH and INESSS), cost-effectiveness is considered in the recommendation (e.g., recommendation for a reduction in price); however, it may not be a driving factor for a list or do not list recommendation. The Netherlands considers cost-effectiveness in combination with burden of disease during the deliberative appraisal. Economic considerations most commonly include a cost-effectiveness analysis for an assessment of population-level value and a budget impact analysis for an assessment of financial feasibility.

Implementation Considerations

Implementation considerations are those related to the adoption of the health technology in practice, such as the organizational requirements, sustainability, financial constraints, or other considerations that may influence the adoption and use of the health technology. The extent to which implementation considerations are discussed in the deliberative framework for a given agency may vary according to jurisdiction depending on the mandate of the HTA body. At NICE, regulations require that NICE recommendations be implemented within 3 months “except when particular barriers to implementation within that period are identified.”¹⁰¹ Consequently, the resource impact or implementation considerations become particularly important to set reasonable recommendations. This agency mandate is different from that of CADTH in Canada, for example, where drug recommendations are issued at a national level and implementation of the recommendation is non-binding for the provinces, territories, and federal government. Payer perspectives may inform implementation considerations, and how this is embedded in the process also varies. It could be used as input into the review reports, presented during the committee meeting, or considered in response to draft guidance. Payer perspectives are important to ensure appropriate implementation, although where and how to embed them in the process is uncertain. Some committees have payer representatives as committee members (e.g., SMC) and others have them as observers or presenters (CADTH). Implementation issues are often considered separate from other factors; however, Thokala and colleagues¹⁰² have suggested embedding resource constraint considerations into the cost-effectiveness analysis. Identification of implementation issues, such as where the health technology may be considered relative to current practice, may also be relevant for pricing or value negotiations with manufacturers. Local implementation considerations have been identified as particularly important for uptake of non-drug technologies in Canada.¹⁰³

Legal Considerations

Legal considerations are not commonly considered explicitly in deliberative, value, or decision-making frameworks. The HTA Core Model and INTEGRATE-HTA framework explicitly include legal considerations as a criterion for HTA assessments. The HTA Core Model indicates that the objective of this domain is to assist HTA producers in identifying the rules and regulations that should be considered when assessing a health technology for implementation. There is limited guidance regarding how legal considerations should be incorporated into the decision-making process, yet there are many points in the HTA decision-making pathway that may be subject to legal considerations. Widrig (2014)¹⁰⁴ offered a framework for legal considerations in HTA from the Swiss perspective using “outside” (macro) and “inside” (micro) perspective considerations to help identify potential legal considerations in the HTA process. Outside legal considerations could include the relationship between the HTA producer and the decision-maker, the responsibility of the decision-maker, and the effects of the decision on the health system and its stakeholders. Inside legal considerations could include those related to the patient, care provider, health technology, financing, and methodology. Methodology is particularly relevant to the deliberative process, and Widrig et al. suggested that it may be helpful for HTA agencies to assess their methodology to identify any potential conflicts with the law.¹⁰⁴

Court challenges for access to health technologies may be more common in jurisdictions where there is a constitutional right to health care.¹⁰⁵ However, court challenges can still occur when it can be argued that restricting patient access to a particular health technology has resulted in discrimination to other parts of the constitution.¹⁰⁵ Legal considerations may overlap with other domains, such as organizational aspects and current health technology use. Some examples of legal considerations are if there is a mandatory vaccination program, a legal obligation to compete the review within a certain amount of time, or an obligation for a certain level of transparency.

Ethical Considerations

Ethical considerations within appraisal frameworks often are less well-defined compared with considerations such as clinical effectiveness and cost-effectiveness.²⁰ Ethical analyses and perspectives within HTA have been relatively limited to date, although recognition of their importance is generally well-accepted. Similar to other domains, the need, weight, and complexity of the ethical analysis depends on the health technology, the role and authority of the HTA organization, time and resources available, and methodological expertise and experience within the organization.³⁹ It has been argued that ethics is integral in the whole HTA process in “how HTA is defined, interpreted, and acted upon.”¹⁰⁶ For example, the requirement for patients to switch from 1 arm of a trial to another¹⁰⁷ may impact the interpretation of the clinical evidence. Likewise, an economic analysis may be conducted with an equity lens to show the impact of the health technology on non-health outcomes such as poverty reduction, or differential cost-effectiveness according to population subgroups. Ethical considerations are embedded in all aspects of the HTA process, and the ethical analysis allows these ethical considerations to become more explicit and visible.³⁹

Environmental Considerations

A scoping review conducted by Polisena and colleagues found that frameworks or methods for environmental assessments of health technologies could be classified into 3 categories: studies that incorporate environmental impact considerations in an HTA (e.g., inclusion of environmental outcomes in an economic analysis from a societal perspective), studies that conduct a synthesis of evidence related to the environmental impact of a health technology, and studies that conduct an environmental assessment of a health technology.¹⁰⁸ The authors recognized that methods and tools for considering the environmental impact of health technologies are available; however, more research is needed to tailor this methodology to assess this impact to support decision-making. Likewise, Marsh and colleagues highlighted the potential value of incorporating environmental data as part of HTA. They recognized that environmental changes have an impact on individuals’ health and that decision-makers often have mandates that extend beyond health care.¹⁰⁹ However, Marsh and colleagues noted that additional research is needed to develop methods suitable for assessing the environmental impact of health technologies. Given the additional effort required to incorporate these considerations, care should be taken in determining in what contexts this additional effort is warranted.¹⁰⁸