CADTH Health Technology Review

Approaches to Evaluations of Virtual Care in Primary Care

Environmental Scan

Authors: Daphne Hui, Bert Dolcine, Hannah Loshak

Key Messages

• A literature search informed this Environmental Scan and identified 11 evaluations of virtual care in primary care health settings and 7 publications alluding to methods, standards, and guidelines (referred to as evaluation guidance documents in this report) being used in various countries to evaluate virtual care in primary care health settings. The majority of included literature was from Australia, the US, and the UK, with 2 evaluation guidance documents published by the Heart and Stroke Foundation of Canada.

• Evaluation guidance documents recommended using measurements that assess the effectiveness and quality of clinical care including safety outcomes, time and travel, financial and operational impact, participation, health care utilization, technology experience including feasibility, user satisfaction, and barriers and facilitators or measures of health equity.

• Evaluation guidance documents specified that the following key decisions and considerations should be integrated into the planning of a virtual care evaluation: refining the scope of virtual care services; selecting an appropriate meaningful comparator; and identifying opportune timing and duration for the evaluation to ensure the evaluation is reflective of real-world practice, allows for adequate measurement of outcomes, and is comprehensive, timely, feasible, non-complex, and non–resource-intensive.

• Evaluation guidance documents highlighted that evaluations should be systematic, performed regularly, and reflect the stage of virtual care implementation to encompass the specific considerations associated with each stage. Additionally, evaluations should assess individual virtual care sessions and the virtual care program as a whole.

• Regarding economic components of virtual care evaluations, the evaluation guidance documents noted that costs or savings are not limited to monetary or financial measures but can also be represented with time. Cost analyses such as cost-benefit and cost-utility estimates should be performed with a specific emphasis on selecting an appropriate perspective (e.g., patient or provider), as that influences the benefits, effects, and how the outcome is interpreted.

• Two identified evaluations assessed economic outcomes through cost analyses in the perspective of the patient and provider. Evidence suggests that, in some circumstances, virtual care may be more cost-effective and reduces the cost per episode and patient expenses (e.g., travel and parking costs) compared to in-person care. However, virtual care may increase the number of individuals treated, which would increase overall health care spending.

• Four identified evaluations assessed health care utilization. The evidence suggests that virtual care reduces the duration of appointments and may be more time-efficient compared to in-person care. However, it is unclear if virtual care reduces the use of medical resources and the need for follow-up appointments, hospital admissions, and emergency department visits compared to in-person care.

• Five identified evaluations assessed participation outcomes. Evidence was variable, with some evidence reporting that virtual care reduced attendance (e.g., reduced attendance rates) and other evidence noting improved attendance (e.g., increased completion rate and decreased cancellations and no-show rates) compared to in-person care.

• Three identified evaluations assessed clinical outcomes in various health contexts. Some evidence suggested that virtual care improves clinical outcomes (e.g., in primary care with integrated mental health services, symptom severity decreased) or has a similar effect on clinical outcomes compared to in-person care (e.g., use of virtual care in depression elicited similar results with in-person care).

• Three identified evaluations assessed the appropriateness of prescribing. Some studies suggested that virtual care improves appropriateness by increasing guideline-based or guideline-concordant antibiotic management, or elicits no difference with in-person care.

Context

The COVID-19 pandemic has accelerated the uptake of virtual care in Canada. In March of 2020, during the first wave of this pandemic, Ontario physicians adapted quickly to the restrictions on in-person care and conducted 52% of care services virtually, including patient appointments, psychotherapy, and consults with other physicians.¹ Just before the first wave in February of 2020, Ontario physicians provided more than 5 million face-to-face services, whereas, during March 2020 (the start of the first wave), Ontario physicians provided more than 3.8 million face-to-face services and approximately 1.1 million virtual services. In April through June 2020, the use of virtual services in Ontario continued to increase, with reports of approximately 2.1 million and more than 2.2 million virtual services conducted in April and June 2020, respectively.¹ Of note, face-to-face services before the COVID-19 pandemic may have consisted of a small number of virtual care services.¹ The Canadian Institute for Health Information reported on the period from March 2020 to March 2021.² The proportion of virtual family physician appointments averaged between 27% and 57% in Alberta, British Columbia (BC), Manitoba, Ontario, and Saskatchewan; specific averages for these provinces were not reported.² In April of 2020, according to the provincial data available, the proportion of virtual family physician appointments were between 40% and 50% for Alberta, between 50% and 60% for Manitoba and Ontario, and around 70% for BC.² In December of 2020, when monthly COVID-19 community cases across Canada were highest during the period from March 2020 to March 2021, the proportion of virtual family physician appointments was approximately 30% for Alberta, about 40% for Manitoba and Ontario, approximately 50% for Saskatchewan, and about 60% for BC.² Adoption of virtual family medical appointments during this period was the highest in BC and the lowest in Alberta.² This rapid pace of adoption also highlighted different issues and challenges associated with delivering care virtually and demonstrated the need for continuous evaluation to help inform strategies and policies, and to develop standards for the effective implementation of virtual care.

While virtual care can be defined in a number of ways, one definition of “virtual care” adopted by the Canadian Medical Association’s Virtual Care Task Force states that it consists of “any interaction between patients and/or members of their circle of care, occurring remotely, using any forms of communication or information technologies, with the aim of facilitating or maximizing the quality and effectiveness of patient care.”^3,4 Virtual care can facilitate the delivery of and access to health care services when barriers ― such as geographic distance or the restrictions due the COVID-19 pandemic ― limit the ability for in-person interactions between patients and health care providers. Virtual visits, also called direct-to-patient care and e-visits, are 1 component of the virtual care model and are defined as “an electronic exchange via videoconferencing, secure messaging, or audio digital tools, where one or more heath care providers deliver health care services to a patient.”⁵ These virtual interactions between health care providers and patients can occur on a synchronous (e.g., videoconferencing, telephone) or asynchronous (e.g., text message) basis.⁶

This literature-based Environmental Scan (ES) will identify methods, standards, and guidelines for evaluating virtual care or completed real-world evaluations to understand how other jurisdictions are approaching virtual care evaluation (with a particular interest in economic evaluation). As virtual care was rapidly and predominantly used during the first wave of the COVID-19 pandemic, most of this review will be in the context of these large shifts related to the pandemic. Identified literature may provide insight into what was considered part of those evaluations (e.g., what inputs, what outcomes, what considerations) to support evaluations of virtual care in Canada. For the purpose of this work, the focus is on interactions between physicians and health teams, and patients, either synchronously or asynchronously, in primary care health settings. Primary care followed the definition of a patient’s first point of contact with the health system, with providers (e.g., physicians and nurses) providing ongoing continuous care including referrals to specialists.⁷ Other digital health interventions or devices (e.g., remote monitoring) are beyond the scope of this scan.

Objectives

The key objectives of this ES are, as follows:

1. Identify methods, standards, and guidelines being used in Canada and other countries for evaluating virtual care (with a focus on economic and outcome-based evaluation), specifically regarding interactions between health care providers and patients.

2. Identify Canadian and international evaluations of virtual care that address interactions between health care providers and patients, and the economic aspect of virtual care.

3. Summarize the methodologies and other relevant information (e.g., inputs, outcomes, measures of value, considerations, lessons learned) from the identified evaluation methods, standards, and guidelines, and completed evaluations.

This ES does not include an assessment of digital health interventions (e.g., e-prescribing) or devices (e.g., remote monitoring) and is focused on virtual care between physicians and health teams, and patients, in primary care settings.

Research Questions

1. With a focus on virtual interactions between health care providers and patients, and outcome-based and economic evaluation, how are jurisdictions in Canada and other countries conducting evaluations of virtual care? What methods, standards, and guidelines inform these evaluations?

2. What completed evaluations of virtual care (that address virtual interactions and the economic aspect) are available in Canada and other countries?

3. What are the parameters used (e.g., inputs, outcomes, measures of value) and what other relevant information (e.g., lessons learned) is considered in evaluating virtual care in the context of virtual interactions between health care providers and patients, and economic evaluation?

Methods

This ES was informed by a limited literature search and will contribute to planned projects to evaluate virtual care in Canada through CADTH’s collaboration with other Canadian research organizations.

Literature Search

A limited literature search was conducted by an information specialist on key resources including MEDLINE, the Cochrane Database of Systematic Reviews, the international HTA database, the websites of Canadian and major international health technology agencies, as well as a focused internet search. The search strategy comprised both controlled vocabulary, such as the National Library of Medicine’s MeSH (Medical Subject Headings), and keywords. The main search concepts were virtual care and COVID-19. No filters were applied to limit the retrieval by study type. Where possible, retrieval was limited to the human population. The search was also limited to English-language documents published between January 1, 2019 and September 1, 2021.

Regular alerts updated the search until project completion; only citations retrieved before October 11, 2021 were incorporated into the analysis.

Screening and Study Selection

One author independently screened titles and abstracts for eligibility according to the inclusion criteria outlined in Table 1. Articles that were published in a language other than English, or were published before January 1, 2019, were excluded. Study selection focused on identifying evaluations of virtual care that fit the Canadian Medical Association’s Virtual Care Task Force definition of virtual care. CADTH acknowledges there are multiple definitions of virtual care that are adopted and/or proposed by various jurisdictions and organizations and committees in Canada (e.g., College of Physicians and Surgeons of British Columbia), but this report follows the Canadian Medical Association’s Virtual Care Task Force definition as it was deemed to be clear and broad, and devised by a pan-Canadian (national) organization. Further, the Canadian Medical Association’s definition is based on the consensus policy dialogue by Shaw et al. (2018) in their report of virtual care policy recommendations for patient-centred primary care, which has been adopted by many committees and organizations.⁴ For the purposes of this ES, literature on digital health interventions (e.g., e-prescribing) or devices (e.g., those used for remote monitoring) were excluded. Moreover, evaluations of virtual care that used survey-based techniques to collect data on perceptions of virtual care were not included, as attitudes and perceptions were considered out of scope for this. However, methods, standards, and guidelines to inform objectives 1 and 3 were considered for inclusion if survey-based techniques to evaluate virtual care were discussed.

Included evaluations focused on primary care, which was defined as the patients’ first point of contact with the health system, with health care personnel (e.g., physicians and nurses) providing ongoing continuous care including referrals to specialists.⁷ Evaluations on specialty care were excluded unless the specialty care was performed in a primary care setting (e.g., opioid treatment programs integrated into primary care settings).⁸ This report focuses on primary care, as the utility and usage of virtual care in primary and specialty care may be different. There are more feasibility and implementation limitations with providing virtual care in certain specialties; namely, virtual care in surgical specialties tend to serve a triage purpose (i.e., to confirm that patients actually require in-person specialized care) with pre- and post-intervention assessments and the surgical intervention being implemented in-person. Virtual care for the purpose of triage may also be used for non-surgical specialized care such as an otorrhea (i.e., drainage of liquid from the ear) performed by an ear, nose, and throat (ENT) specialist.⁹ Additionally, there are different considerations and models of providing care across various medical specialties and some specialties can be more easily implemented with virtual care such as psychiatry. Further, patients who require specialized care (i.e., tertiary or quaternary care) may receive chronic and more multidisciplinary health care (e.g., treatment for physical health of various health contexts [e.g., comorbidities], mental health, and rehabilitation). Therefore, this ES will specifically focus on evaluations of virtual care in primary care. However, identified methods, standards, and guidelines for evaluations of virtual care were not limited to primary care, as most of these documents informed evaluations of virtual care in general and were not specific to a practice type (e.g., hospital or private clinics), level of health care (e.g., primary care), health context, or disease.

Synthesis Approach

Findings from the literature search were summarized narratively. When summarizing the identified evaluations of virtual care, relevant findings were separated based on outcome type (economics, health care utilization, participation, clinical outcomes, and appropriateness of prescribing). When summarizing publications informing methods, standards, and guidelines for evaluations of virtual care, individual findings were separated based on the reporting organization (e.g., American Medical Association, Heart and Stroke Foundation of Canada). Of note, the publications informing methods, standards, and guidelines for evaluations of virtual care will be referred to as evaluation guidance documents for the remainder of this report. Namely, information from the Heart and Stroke Foundation of Canada was summarized generally and not specific to stroke management and cardiovascular prevention and rehabilitation, which were the focus of these publications. The Findings sections for Objectives 1 and 2 report on the relevant information and evidence for each individual publication while the Findings section pertaining to Objective 3 summarizes the overall findings of the included information and evidence collectively. Included literature referred to virtual care with various terms (e.g., telehealth, e-visits, teleconsultation, telemedicine, remote consultations), but the terms “virtual care” or “virtual consultations” were used throughout this ES for consistency. Similarly, included literature referred to the comparators with different terms (e.g., face-to-face, in-office, pre-protocol implementation); however, the term in-person was used throughout this ES for consistency except when specifically referring to Li et al. (2021) and Han et al. (2020). ^10,11 These evaluations referred to the comparator as face-to-face visits; although the authors did not specify that these were in-person visits, other information provided in the articles suggested that the face-to-face visits were predominantly or entirely conducted in person, which supported their inclusion in this ES. Namely, Li et al. (2021) noted that face-to-face consultations largely facilitated visits before the COVID-19 pandemic, with the comparator group referred to as remote consultations, and Han et al. (2020) noted that video modalities were included in the remote consultation group (comparator of the face-to-face group).^10,11 The virtual care (intervention) and comparator terms used in the source publications of the included evaluation guidance documents and evaluations are detailed in Table 4 and Table 5, respectively. Extracted information from the evaluation guidance documents were paraphrased and summarized with the document-specific terminology and distinct statements reported in italics or with quotations and references to the page number from the source document.

Opportunities for Stakeholder Feedback

Stakeholders (e.g., clinicians, patient groups, and other interested parties) were given the opportunity to provide feedback on the draft report. The draft report was publicly posted on the CADTH website for 4 weeks. Comments received were considered in the final version.

Findings

The findings presented are based on the literature search results. The literature search yielded 416 citations from which 110 were selected for full-text screening and 11 publications were determined to be eligible. Of these, 3 were systematic reviews (SRs)^11-13 and 8 were non-randomized comparative studies.^8,10,14-19 For additional information, 7 publications from the grey literature search^20-26 were also included. Altogether, 18 publications were included to address the research questions; of these, 7 were evaluation guidance documents being used in various countries to evaluate virtual care in primary care settings and 11 were evaluations of virtual care in primary care settings. Overall, the evaluation guidance documents recommended that evaluations measure and consider the effectiveness and quality of clinical care including safety outcomes, time and travel, financial and operational impact, participation outcomes (e.g., attendance, no-show rates), health care utilization (e.g., duration and frequency of appointment, need for follow-up), technology experience including feasibility, user satisfaction (e.g., patients and clinicians), and barriers and facilitators or measures of health equity. Identified evaluations suggested that in some situations virtual care may be more cost-effective than in-person care and reduces the cost per episode, patient expenses (e.g., travel and parking costs), and the duration of appointments per patient. However, virtual care may increase the total number of appointments because of the reduced duration of each appointment, thereby allowing additional time to conduct more appointments. This could increase overall health care spending and utilization. Additionally, it is unclear if virtual care reduces the use of medical resources and the need for follow-up appointments (provided by the same clinician or type of care to address unresolved or additional health concerns not adequately addressed in the initial consultation), hospital admissions, and emergency department visits. Evaluations of virtual care assessing participation outcomes, clinical outcomes in various health contexts, and appropriateness of prescribing reported variable findings.

Objective 1: Identify methods, standards, and guidelines being used in Canada and other countries for evaluating virtual care (with a focus on economic and outcome-based evaluation), specifically regarding interactions between health care providers and patients.

Seven evaluation guidance documents were identified from the grey literature search.^20-26 Among these, 2 were published by a Canadian group (Heart and Stroke Foundation of Canada), 2 were published by American groups (American Medical Association and the National Quality Forum [NQF]), 1 was published by a UK group (UCLPartners), 1 was published by an Australian group (Queensland Health), and 1 was published by the Pan American Health Organization (PAHO; greatest representation was from Mexico and Colombia, followed by Peru; there was no representation from Canada). The Heart and Stroke Foundation of Canada published implementation documents specifically for virtual stroke management in 2020;²¹ and for virtual cardiovascular prevention and rehabilitation in collaboration with the Canadian Association of Cardiovascular Prevention and Rehabilitation (CACPR) group of multidisciplinary expert reviewers in 2021.²² The American Medical Association published a framework for measuring the value of digitally enabled care through the measurement of value streams from the review of existing literature and interviews with 20 national experts.²⁵ The NQF is an American nonpartisan, not-for-profit health care organization that published a report focused on the development of a measurement framework based on an ES to guide virtual care measurement priorities and their impact on health care delivery and outcomes.²⁶ UCLPartners is an academic health science partnership of National Health Service (NHS) providers and universities in the UK. The UK UCLPartners devised a guide to evaluate non‒face-to-face clinics.²⁰ The Australian Queensland Health published an evaluation resource guide in 2016.²⁴ PAHO published a summary of discussions focused on “defining evaluation indicators for telemedicine projects” that took place in April and May of 2015.²³ Table 4 (Appendix 1) summarizes the main characteristics of these publications. A summary of findings of these individual publications follows.

American Medical Association

The American Medical Association published a framework devised of environmental variables and value streams. The environmental variables impact each value stream and value streams are measured to determine how virtual care generates value.²⁵ The authors reported on 5 environmental variables including type of practice, payment arrangements and rates, social determinants of health of patient population, clinical use case, and virtual care modality. The authors identified 6 value streams: clinical outcomes, quality, and safety; access to care; patient, family, and caregiver experience; clinician experience; financial and operational impact; and health equity. The environmental variables and value streams and associated sub-streams are detailed herein. Examples of measurements for the respective value streams are also reported herein; however, this is not an exhaustive list of the proposed measurements detailed in the American Medical Association document.

Environmental Variables²⁵

• Type of practice (e.g., independent practice): impacts scope of clinical programs, prioritization of clinical cases, resource availability, and potential scalability

• Payment arrangements and rates (e.g., fee-for-service): impacts priorities from a business perspective and financial sustainability

• Social determinants of health of patient population (e.g., access to broadband and technology, race and ethnicity, income): patient demographics associated with health inequities impacting access to, and benefits from, virtual care, which includes connectivity requirements and choice of device and platform

• Clinical use case (e.g., primary care): impacts how care is implemented, technology and data collection requirements, and business necessities

• Virtual care modality (e.g., video visit, virtual secure messaging, remote patient monitoring ― not focused on in this ES): impacts technology cost and requirements, operational requirements, and how the provider is paid

Value Streams²⁵

1. Clinical outcomes, quality, and safety

• Clinical quality and safety outcomes:

  • measures of mortality, functional status (e.g., Functional Independence Measure), disease morbidity (e.g., Patient Health Questionnaire-9), and Healthcare Effectiveness Data and Information Set (e.g., HEDIS)

  • patient-reported outcomes (e.g., Brief Pain Inventory)

  • emergency department visits

  • number of visits required for correct diagnosis

  • rates of antibiotic prescribing, adverse events, and readmission

• Clinical processes:

  • improvement in detection of disease

  • adherence to medication, treatment plans, discharge instructions, and evidence-based guidelines

2: Access to care

• Availability of care:

  • median travel time to care, per patient, and time to third-next available appointment or consultation

  • percentage of patients who completed a specialty referral within 14 days of referral and with coverage for virtual visits with current insurance

  • number and frequency of patient appointments with clinician and/or care team

  • reduction in patient transfers

• Equitable care:

  • percentage of patients who can receive virtual care in their desired language, who delay virtual care as a result of barriers to access (e.g., lack of access to technology or internet connection), and with disabilities and are able to receive virtual care through adaptive technologies

  • patients’ out-of-pocket costs based on a percentage of their household budget

3: Patient, family, and caregiver experience

• Clinical and/or technology experience:

  • net promoter score

  • patient activation measure

  • hospital or clinician and group Consumer Assessment of Healthcare Providers and Systems Survey (HCAHPS or CG-CAHPS)

  • reported understanding of physicians’ instructions by assessing patients’ understanding of treatment instructions

4: Clinician experience

• Technology experience:

  • reported ease with using the technology to facilitate virtual care

• Work experience:

  • percentage of visits conducted virtually versus in person

  • annual percentage of annual recruiting costs or physician turnover

  • self-reported engagement and satisfaction with work

  • American Medical Association‒recommended physician satisfaction surveys (e.g., Mini Z Burnout Survey)

  • duration of visit versus an equivalent in-person visit

5: Financial and operational impact

• Direct revenue:

  • payment for technical services and professional services

  • performance-based payments

  • revenue from fees paid by other health centres to participate in virtual care services (e.g., fees that community hospitals pay to participate in a telestroke network)

• Indirect revenue:

  • number of new patients acquired

  • patient retention rate

  • increased bed or appointment availability

  • percentage of referrals completed

• Direct expenses:

  • clinical care and malpractice expenses

  • expenses related to implementing virtual care (e.g., set-up and maintenance costs, IT infrastructure, staffing)

  • “Total cost per episode of care or per member per month (for payers, employers, and clinicians in risk-based arrangements)” (p. 16)²⁵

• Operational efficiencies:

  • number of individual patients under the care of a specific provider (i.e., clinician panel size)

  • no-show rate

  • length of stay and inpatient rate or rate of emergency department use

6: Health equity

• Equity in clinical outcomes, quality, and safety

• Equity in access to care

• Equity in patient, family, and caregiver experience

• Equity in clinician experience

• Equity in financial and operational impact

Heart and Stroke Foundation of Canada

The Heart and Stroke Foundation of Canada noted that considerations for planning for evaluations included accessibility, effectiveness, efficiency, system integration and continuity, patient and provider experience, and technical efficiency and responsiveness.^21,22 Further, it was specified that evaluations should include an assessment of the overall program and individual virtual care sessions, with overall program evaluations addressing established quality indicators and standards of care for a specific health disease or context (e.g., cardiovascular rehabilitation). Additionally, data collection for evaluations was suggested to be integrated into patient health care records and, when possible, utilize existing databases and registries. It was also noted that data should be reviewed and summarized regularly (e.g., monthly review summarized into an annual report). It is recommended by the Heart and Stroke Foundation of Canada to investigate adherence to clinical guidelines and performance measures to assess patient improvement, quantify quality of care, identify areas of improvement, determine program efficacy and effectiveness (e.g., cost savings, effect on patient outcomes, recurrence of clinical events, and hospital readmissions) and accessibility and barriers (e.g., extent of access, completion rates, and wait times). The planning considerations and consensus- and evidence-based performance measures for individual sessions should be specific to a diagnosis or health context and are detailed here.

Considerations^21,22

• Accessibility: wait times, need for transfers to more advanced care, and degree of access to care

• Effectiveness: impact of access to specialized services through virtual health care based on patient outcomes, length of stay, complications, readmissions, and recurrent disease (e.g., recurrent stroke)

• Efficiency: timely access to virtual health care services, cost savings, and streamlining health services

• System integration and continuity: degree to which virtual care allows for continuity of care, follow-up care, and a smooth transition from traditional care to virtual care

• Patient experience: patients’ perceptions of the virtual care appointment

• Provider experience: health care provider’s perceptions of the virtual health care appointment and providers’ competency level and educational preparation to participate in virtual care

• Technical efficiency and responsiveness: degree to which technology is functional without technical difficulties that could elicit a negative impact

Performance Measures^21,22

• Sub-set of current performance measures to determine appropriateness and effectiveness for evaluation of virtual care

• Degree to which health care providers can perform required assessments and provide required treatments through virtual care sessions (e.g., ask providers if they feel that the virtual session allows for a thorough or complete assessment)

• Median frequency and duration of virtual care sessions; frequency of negative events (e.g., percentage of patients receiving inappropriate consultation and treatment); frequency of positive events (e.g., percentage of patients receiving appropriate consultation and treatment); and attendance data

• Median wait times for consultation, proportion of patients seen within targeted times (e.g., based on medical history and urgency of symptoms), time to initiation of consultation from various benchmarks (e.g., symptom onset, arrival at medical centre, and completion of tests such as imaging)

• Effectiveness of virtual care versus in-person sessions

• Percentage of patients requiring in-person follow-up for assessments that could not be addressed virtually

• Percentage of sessions that involve technical difficulties affecting the quality of the session and ability to provide health services

• Measure of independence at discharge (e.g., Median Rankin and National Institutes of Health Stroke Scale scores) and location of discharge (e.g., personal residence)

• Travel distance and cost saved by the patient

• Health care provider’s rating of virtual appointments and willingness to expand virtual care in their practice

• Patient’s rating of quality of virtual care appointments and willingness to participate in future sessions

UCLPartners

UCLPartners noted that the evaluation should provide evidence alluding to the impact on outpatient pathways, clinical safety, cost, and breadth of implementation.²⁰ Considerations included identification of the stakeholders (e.g., users and decision-makers) during the planning of the evaluation; selection of an appropriate comparison to assess the impact or improvement; and practical issues such as timing of the evaluation, data accessibility for collection and analysis, and sufficiency of resources. The authors suggested various quantitative and qualitative outcomes when evaluating virtual care, which are detailed here.

Quantitative Outcomes²⁰

• Number of patients treated within virtual and in-person care

• Proportion of patients converted from in-person to virtual care

• Waiting times and waiting list before and after virtual care implementation

• Number of virtual and in-person appointments, follow-up attendances, and referrals received

• Outcome of subsequent care (e.g., discharge to community or referrals to specialists)

• Costs

• Time spent by provider attending to patients in virtual and in-person appointments

• Involvement of other practices and other community settings

Qualitative Outcomes²⁰

• Providers’ experiences with offering virtual care such as the quality of care administered or convenience

• Patient acceptability or satisfaction

Of note, qualitative outcomes may be evaluated through surveys, focus groups, or interviews.

National Quality Forum

The National Quality Forum devised a measurement framework to guide virtual care evaluation with 4 domains and associated subdomains and measurement concepts.²⁶ The main domains included access to care, financial impact or cost, experience, and effectiveness. Travel, timeliness of care, actionable information, added value of virtual care to provide evidence-based best practices, patient empowerment, and care coordination were identified as measurement areas of highest priority. The domains, subdomains, considerations, relevant measurement category, and potential measure concepts are detailed in Table 2.

Pan American Health Organization

The Pan American Health Organization suggested that evaluations should demonstrate the benefits of virtual care compared to health services delivery with the health impact measured.²³ The Pan American Health Organization noted that evaluations of virtual care should be conducted at each phase of implementation and involve assessments of individual components (e.g., sessions) and overall assessments. Virtual care evaluation was suggested to involve systematic methodology, with a focus on feasibility and acceptability. Further, the evaluation model should represent conditions that are similar to the in-person modality and evaluations should regularly assess indicators and consist of regular audits. In the early phases, evaluations were recommended to focus on attaining the initial objectives and on how the strategies are aligned with the targets. During the mid-stages and established phases, evaluations should focus on optimization and social and economic benefits, respectively.

The Pan American Health Organization stated that a variety of considerations should be prioritized for the evaluation: macro aspects including economic, legal, political, and sociocultural considerations and micro aspects including strengths, weaknesses, and financial considerations (this list is not exhaustive). Additionally, components of the health sector should be considered including primary health care; hospital services; rehabilitation; barriers; acceptability to patients and community; and role of academia, medical associations, and regulators. Therefore, pivotal objectives would include project feasibility; acceptance by professionals; sensitization of authorities and decision-makers; technical capabilities of user; cost, benefits, efficiency, and effectiveness; and accessibility and acceptability both by the patient and by health care providers.

Proposed evaluation indicators were categorized based on the stage of implementing virtual care (short-, medium-, long-term), measurement relevance (timeliness, effectiveness, quality, efficiency, endogenous, and exogenous). From these, a list of primary indicators were devised and are detailed in Table 3 (program coverage, hours available for virtual consultations with specialists, program not operating because of technical issues, virtual consultations held [successful virtual consultations], transfers generated through virtual care, subsequent virtual care appointments [based on the first appointment], wait time for virtual consultation, specialty consultations by virtual care, and patient satisfaction). It was highlighted that the definitions of endogenous and exogenous indicators need to be clarified. Further, the primary indicators were categorized as measures of quality, performance, impact, effectiveness, and demographics. It was emphasized that each health context may necessitate additional specific service indicators and economic indicators, and that indicators should be clear, measurable, comparative, and have a certain frequency of measurement. Overall, all proposed indicators evaluate a component of access, user satisfaction, performance, quality of services offered, and cost benefit, with the perception of quality, benefit, and access noted to be the best indicators to evaluate virtual care. The Pan American Health Organization specifically highlighted that indicators informed by the framework of health economics should be considered including cost-benefit and cost-utility estimates such as out-of-pocket expenditures and evaluation of price and quality. It was also highlighted that savings did not have to come from a cost perspective but could be represented by a reduction in treatment times using virtual care.

Evaluation Indicators Categorized Based on Chronological Impact²³

Short-term:

• Number of virtual consultations that occurred in a given period

• Increase in virtual consultations conducted (measure of progress)

• Patient savings (measure of financial impact on patients)

• Wait time between scheduled time and the virtual care appointment

• Time from when a virtual consultation is requested until it is conducted (measure of virtual care’s advantage over regular referrals and time saved)

• Satisfaction questionnaire

• Increase in number of specialties per available unit (measure of acceptance among physicians and hospital administrators)

Medium-term:

• Hospitals participating in virtual care in reference to the national total

• Relationship between virtual consultations performed and the relative decrease in regular consultations (i.e., in-person)

• Number of free hours

• Number of technical problems per unit

• Time required to resolve technical problems per unit

• “Number of differed teleconsultations” (p.15)²³

Long-term:

• Improvement in monthly consultations over previous years

• Number or percentage of patients being monitored by virtual care

• Average savings over the previous year

Evaluation Indicators Categorized Based on Measurement Relevance²³

Timeliness indicators:

• “Number of patients who need to be transferred/number of patients transferred

• Number of physicians in the remote unit/number of physicians trained last year

• Number of medical specialties in the remote unit/number of medical specialties in the referral unit

• Transfer time to the remote unit/transfer time to the referral unit” (p.17)²³

Effectiveness indicators:

• Number of patients seen after the virtual care was implemented compared with the number of cases before virtual care was implemented for a given period (i.e., number of appointments conducted before and after virtual care was implemented)

• Number of accessible specialists after the virtual care was implemented compared with the number before virtual care was implemented for a given period

• Number of individuals with disabilities or physical, economic limitations that were able to access specialized care through virtual care services

Quality indicators:

• Number and range of specialists adopting virtual care

• Stability and reliability of the virtual care system and method used to transmit information

Efficiency indicators:

• Monitoring costs

• Access

• Coverage

• Patient perception

Endogenous indicators:

• Coverage

• Trained technical personnel

• Necessary equipment and supplies

• Patient-reported experiences

Exogenous indicators:

• Costs of medical care

• Medical effectiveness

• Service access

Proposed Primary Indicators²³

Table 3: Proposed Primary Evaluation Indicators by the Pan American Health Organization²³

.

Queensland Health

Queensland Health reported the following dimensions of service performance relevant to virtual care evaluations: access, effectiveness (e.g., health outcomes; safety and quality; acceptability, satisfaction, and appropriateness in the patient perspective; acceptability and adoption in the health professional perspective; and technology and functionality), and efficiency and service outcomes (e.g., input per output produced such as costs or resources consumed and service outputs and value generated for the health of the community).²⁴ Allied health teams often set the goal of implementing virtual care as efficiency (quantity of outputs in relation to quantity of inputs) and cost savings for the service, particularly regarding expenses and travel time. The authors highlighted that perspective is very important for evaluations that have an economic component. Perspective determines which inputs (costs) and outputs (benefits or effects) are included and how the outcome is interpreted. Costs are not limited to out-of-pocket expenses but can also include time spent travelling for patients or staff and the impact on patients and their families such as the need to take time off work or to arrange childcare. However, implementing virtual care can transfer costs and benefits from 1 group or entity to another, which complicates evaluations. Thus, the selection of the appropriate perspective requires the following considerations: the degree of the flow of costs and benefits between the allied health team, other parts of the health service, other agencies, or patients relevant to the evaluation. The authors also listed several published evaluation frameworks that can be used to consider various factors of a virtual care evaluation such as clinical and health, technology, socioeconomic, environmental, and service and organizational (see p.26 of the source document) factors.²⁴

The authors highlighted that the focus of the evaluation is determined by the stage of virtual care implementation:

• pre-introduction stage ― technical and financial feasibility, service need, preparedness of staff and patients, and legal or ethical issues

• pilot/trial stage ― clinical effectiveness, legal and ethical issues, technical feasibility, service process (administrative or operational), and safety and quality

• adoption and implementation ― clinical effectiveness, health outcomes, service efficiency, safety and quality, and acceptability and satisfaction

• established and translational stage ― health outcomes, acceptability and adoption, access, efficiency, and technology (scalability).

Additionally, the authors highlighted the following key considerations: What decisions will be informed by the results? Who will be the primary users of the evaluation results? What are the outcomes that are to be achieved? What opportunities may be developed by the evaluation? How does the evaluation fit into existing evidence available for virtual care? It was noted that secondary objectives should be devised to anticipate related questions that may arise from the primary findings. Further, Queensland Health highlighted key decisions (in bold) and associated considerations for virtual care evaluations, which are detailed next.

Key Decisions and Associated Considerations²⁴

Which virtual care service will be evaluated?

• Appropriately limit scope for a less complicated evaluation and to make the implementation faster and less resource-intensive

• Need to ensure that the limited scope does not skew evaluation outcomes and is reflective of clinical practice (e.g., limiting the population by disease severity or demographic factors such as age)

Which service will be used as a comparator?

• Select a pre- and post-comparison of a single population if there are no major additional changes that occur between the 2 time comparisons

• Select a comparator devised of different locations or staged implementation if there are no major differences between clinical populations

• Select a comparator based on standard care versus virtual care (simultaneous subgroup comparison [e.g., randomized controlled trial]) if there are no major ethical issues with allocating patients to the different groups of care

When will the evaluation occur — timing?

• Need to consider specific considerations of particular stages of virtual care implementation (e.g., data collection during the early stages when providers and patients are still familiarizing themselves may skew the results)

• Need to consider specific considerations of particular data collection periods (e.g., seasonal demands may impact findings through differing resourcing needs such as availability of staff)

• Need to consider strategic or organizational factors (e.g., additional funding)

How long will the evaluation occur — duration?

• Duration should facilitate collection of a sufficient amount of data required for meaningful findings

• Duration should appropriately represent real-world clinical practice (longer data collection periods are less likely to be influenced by timing)

• Duration should allow for adequate measure of outcomes (time frames required to measure the outcome appropriately and comprehensively)

Which outcome measures will be used?

• Select outcomes that are associated with the following characteristics:

  • Feasible (whether measurement is possible given limitations such as time, resourcing, capacity, and skills of providers)

  • Valid (whether measurement addresses the objectives of the evaluation)

  • Reliable (whether the measurement is accurate)

  • Attributable (whether the measurement reflects a plausible and potential effect)

  • Comparable (whether the outcome supports a relevant and appropriate comparison)

  • Sensitive (whether there is a realistic chance of demonstrating an effect)

What is a meaningful evaluation outcome?

• Outcome that is related to the target or goal of the evaluation

• Outcome measurement that can demonstrate change required to indicate success of virtual care

• Outcome reflects the need of relevant stakeholders (e.g., patients and their family, health care providers, broader health system, and local community or entire society such as a country)

Objective 2: Identify Canadian and international evaluations of virtual care that address interactions between health care providers and patients, and the economic aspect of virtual care.

Eleven evaluations of virtual care in primary care were identified; of these, 3 were SRs^11-13 and 8 were non-randomized comparative studies,^8,10,14-19 of which 6 of 8 were retrospective medical record reviews and the remaining 2 involved prospective data collection (1 study was a survey-based study but only the participation findings were extracted for this ES and the other was a database study). Eight articles were published in the US, 2 articles were published in the UK, and 1 article was published in Australia. All the non-randomized comparative studies (retrospective or prospective) included data specific to a period of COVID-19 or compared pre-COVID-19 to a COVID-19 period for the in-person care versus virtual care comparison. Evaluations focused on primary care in general antibiotic prescriptions (by a physician), mental health services, and drug addiction services. The identified evaluations assessed the following measures of value: economics, health care utilization, participation, clinical outcomes, and appropriateness of prescribing (for evidence specific to prescriptions). Table 5 in the Appendix summarizes the main characteristics of the included evaluations and the following findings are summarized based on the aforementioned measures of value.

Evaluations Reporting Economics and Health Care Utilization Outcomes

Identified evaluations of virtual care with a focus on the economic aspect were limited to 2 SRs,^12,13 while 4 studies assessed health care utilization.^11-13,18 Carrillo de Albornoz et al. (2021) performed an SR focused on primary care including mental health and allied health services.¹² The authors reported that virtual care reduced health care cost and utilization among various appointment types within primary care (e.g., mental health and smoking cessation) and allied health services (e.g., speech therapy and postpartum care). Cost analyses, identified by this SR,¹² demonstrated that virtual consultations are more time- and cost-efficient compared to in-person primary care; namely, 1 study on speech therapy demonstrated that video conferencing was associated with significantly lower service cost per patient and significantly reduced the number and duration of appointments.¹² Increased cost savings were primarily due to reduced travel and parking costs.¹² Further, 1 study that included primary care physicians, retail health clinics, emergency departments, and urgent care centres noted that video consultations reduced health care utilization (e.g., imaging and lab tests); however, the number of follow-up visits ― within 3 weeks ― and cost per episode (e.g., pharmacy and medical costs) were similar compared to in-person appointments.¹² Another study found that virtual consultations were shorter on average ― less than 10 minutes ― compared to in-person visits, which averaged 15 to 30 minutes, among appointments providing standard post-natal care within primary care centres.¹² Overall, this SR¹² suggested that virtual consultations reduce the cost per episode of care but may increase the number of individuals treated, which would increase overall health care spending.¹² Nguyen et al. (2021) performed an SR that included studies conducted in outpatient or inpatient settings but did not specify the level of care (i.e., primary care).¹³ Most studies in this SR¹³ demonstrated that virtual care was associated with lower treatment costs but resulted in variable effects on health care utilization. Phillips et al. (2021) evaluated the comparative effectiveness of virtual care to in-person care in a respiratory assessment centre and found no statistical differences between virtual care versus in-person visits for hospital admissions (2.0% versus 2.8%, respectively), emergency department visits (5.0% versus 3.9%, respectively), or follow-up virtual care appointments (11.0% versus 9.7%, respectively) within a 14-day window.¹⁸ Patients with an initial in-person visit had a higher percentage of related hospital admissions (13.0% versus 7.5% in the in-person versus virtual care groups, respectively) and higher related follow-up virtual care visits (44.7% versus 42.5% in the in-person versus virtual care groups, respectively) within a 14-day window.¹⁸ Han et al. (2020) performed an SR focused on antibiotic prescribing among primary health care settings and reported on studies that investigated follow-up visit rates after initial consultation for the same presentation but evidence was mixed regarding whether remote consultations were more likely to be followed up with another consultation for the same condition.¹¹

Evaluations Reporting Participation Outcomes

Two studies found that virtual care reduced attendance rates.^12,14 Chakawa et al. (2020) evaluated the comparative effectiveness of virtual care to in-person care in an integrated primary care setting at an inner city pediatric clinic and found that attendance rates for integrated primary care visits using virtual care were significantly less than attendance rates for in-person visits among the overall sample (standardized mean difference = 0.41, P < 0.01) and matched sample (standardized mean difference = 0.56, P = 0.01).¹⁴ Odds of non-attendance were approximately 4 times greater than attendance for virtual care visits among the overall sample even when accounting for other variables such as familiarity with the primary care provider.¹⁴ One study in the SR by Carrillo de Albornoz et al. (2021) reported that the telephone group had fewer sessions attended among smoking cessation counselling.¹²

Two studies found that virtual care improved attendance rates.^8,16 Frank et al. (2021)¹⁶ evaluated the comparative effectiveness of virtual care to in-person care in an academically affiliated primary care clinic with integrated mental health services staffed by psychologists and psychology trainees. The authors found that after implementing virtual care for mental health services (mean ± standard deviation [SD] = 2.17 ± 4.36), the number of appointments attended significantly increased compared to in-person services (mean ± SD = 1.19 ± 2.08) (P = 0.002).¹⁶ There was also a significant decrease in the number of cancellations (patient or clinician initiated) during the virtual care services time period (mean ± SD = 0.14 ± 0.49) compared to the in-person time period (mean ± SD = 0.53 ± 1.03) (P < 0.001), but no significant difference was found in the number of no-shows across the 2 periods.¹⁶ An additional analysis was performed on a subgroup of individuals who attended more than 1 appointment during the in-person period (March to December 2019) or virtual care period (March to December 2020); comparing in-person to virtual care demonstrated a significant increase in attendance (P = 0.002) and significant decrease in cancellations (P < 0.001) with virtual care but no difference in no-shows.¹⁶ O’Gurek (2021) conducted a virtual care evaluation in an outpatient opioid treatment program within the Department of Family and Community Medicine at a university medical centre in an urban setting.⁸ O’Gurek (2021) found that the no-show rates significantly reduced with virtual care (P ≤ 0.05), but there were no differences between show rates.⁸

Drerup et al. (2021) also assessed no-show rates in a primary and specialty care clinic.¹⁵ The authors found that virtual care significantly reduced patient no-show rates. Virtual care appointments exhibited a significantly lower no-show rate (7.5%) compared to in-person visits (36.1%) during the same period (March 16 to May 1, 2020) (P < 0.0001) and the non‒COVID-19 (January and February, 2020) in-person no-show rate (29.8%) (P < 0.0001).¹⁵ One study assessed the completion rate across practice types.¹⁷ Gmunder et al. (2021) found that primary care (66.06%) had a lower completion rate than surgical specialties (68.60%) and other specialties (e.g., optometry, audiology, exercise physiology) (69.60%) but higher than medical specialties (presumably non-surgical medical specialties) (60.89%) in a medical network consisting of a hospital, outpatient clinics, a cancer centre, and an eye-specific hospital.¹⁷ Of note, all patients in this study received an automated appointment reminder before the visit via a short message service, (SMS) text message or phone call, based on their preference.¹⁷ One study assessed adherence to virtual care treatment and discontinuation rates.¹² The SR by Carrillo de Albornoz et al. (2021) identified 1 randomized controlled trial on post-natal care that reported twice as many participants discontinued care in the virtual consultation (25%) versus in-person groups (12.5%) with the main reasons being due to the inability to attend the final follow-up visit and technical issues (virtual consultation group, only). Included studies in the SR¹² evaluating care for mental health demonstrated conflicting findings, with some reporting higher or lower discontinuation rates in-person compared to virtual consultations; however, 1 study reported that attrition was significantly lower in primary care patients receiving cognitive behavioural therapy over the telephone compared to in-person care (20.9% versus 32.7%, P = 0.02).

Evaluations Reporting Clinical Outcomes

Three studies assessed clinical outcomes that were specific to the disease or health context.^12,13,16 Frank et al. (2021) reported on a study performed in a academically affiliated primary care clinic with integrated mental health services and evaluated clinical outcomes using Clinical Global Impressions Scale scores.¹⁶ Among participants who had appointments in person (March to December 2019) and virtually (March to December 2020), there was a significant decrease in symptom severity scores (Clinical Global Impressions-S) from in-person appointments (mean ± SD = 3.61 ± 0.70) to virtual care appointments (mean ± SD = 3.33 ± 0.97) (P = 0.020) and significant improvement in the improvement scores (Clinical Global Impressions-I) from in-person (mean ± SD = 3.06 ± 0.87) to virtual care appointments (mean ± SD = 2.44 ± 0.51) (P = 0.002).¹⁶ This suggests that virtual care facilitated a decrease in symptom severity, based on Clinical Global Impressions-S scores, and improved symptoms, based on Clinical Global Impressions-I, compared to in-person appointments.¹⁶ However, it is important to note that this may be attributed to the number of sessions attended and not necessarily the use of virtual care over in-person care.

The SR by Carrillo de Albornoz et al. (2021) noted that consultations via telephone and videoconference were as effective as in-person visits in improving clinical outcomes, specific to the disease or health context, in primary care and mental health. Evidence for smoking cessation counselling was variable, as 1 study demonstrated non-inferiority with this rate and another study found an increased continuous abstinence rate.¹² Evidence for depression was also variable; 1 study reported that telephone-delivered cognitive behavioural therapy in primary care patients with depression was inferior to in-person therapy by the 6-month follow-up, but another study found non-significant differences at any time up to the 6-month follow-up in patients with depression and alcohol abuse. Another study assessing depression among older veterans found no significant differences in patient satisfaction and quality of life as measured by the Short Form Health Survey-36 items. Further, some included studies of this SR¹² compared videoconferencing to telephone-delivered interventions and reported similar effectiveness to reduce depression symptoms at various points of follow-up. The SR performed by Nguyen et al. (2021) reported comparable clinical outcomes to in-person visits but did not specify the level of care (i.e., primary care).¹³

Evaluations Reporting Prescription Outcomes

Three articles investigated antibiotic prescribing using virtual modalities.^10,11,19 Ray et al. (2021) compared in-person care and virtual care using guideline-concordant antibiotic management for acute respiratory tract infections among pediatric patients.¹⁹ Across the 6-month period, guideline-concordant antibiotic management occurred in 92.5% of virtual care visits and 90.7% of in-person visits (P = 0.004). Over time, guideline-concordant antibiotic management during virtual care visits increased from 88% to 97% between April and September 2020.¹⁹ Han et al. (2020) performed an SR that also reported on guideline-concordant prescription rates or guideline-recommended prescribing rates (against local or national guidelines of the US) from 4 observational studies. Guideline-concordant antibiotic management for sinusitis and urinary tract infection demonstrated no significant differences between remote and face-to-face consultations. However, conflicting results were reported for acute respiratory infection.¹¹ One included study of the SR performed by Carrillo de Albornoz et al. (2021) reported that virtual consultations delivered by videoconferencing had significant improvements in guideline-based antibiotic management but elicited less appropriate testing and increased the number of follow-up visits.¹²

Li et al. (2021) evaluated the appropriateness of prescribing with compliance to the recommendation of the clinical scale scores Centor or FeverPAIN in tonsillitis, for which the scoring system usage rate was significantly higher in the remote consultation group compared to the face-to-face group (P = 0.0415).¹⁰ During remote consultation, 51 out of 67 (76.1%) prescriptions complied with the Centor and FeverPAIN recommendation and were deemed appropriate, but 16 out of 24 (66.7%) did not comply and were inappropriate. During face-to-face consultations, 16 out of 67 (23.9%) prescriptions adhered to the recommendation, but 8 out of 24 (33.3%) did not comply and were inappropriate. Differences in appropriate and inappropriate prescriptions between remote consultation and face-to-face groups were not statistically significant.¹⁰ Of note, what constituted remote and face-to-face appointments were not reported; thus, it is presumed that remote refers to virtual care and face-to-face care were in-person appointments.

Objective 3: Summarize the methodologies and other relevant information (e.g., inputs, outcomes, measures of value, considerations, lessons learned) from the identified evaluation methods, standards, and guidelines, and completed evaluations.

Summary of Included Evaluation Guidance Documents

Queensland Health noted that outcomes must be feasible (whether measurement is possible given limitations such as time, resourcing, and capacity and skills of providers), valid (whether measurement addresses evaluation objectives), reliable (whether measurement is accurate), attributable (whether measurement produces a plausible and potential effect), comparable (whether the outcome supports a relevant and appropriate comparison), and sensitive (whether there is a realistic chance of showing an effect).²⁴ Additionally, Queensland Health suggested that meaningful evaluations include outcomes that are related to the target or goal of the evaluation, reflect the needs and priorities of relevant stakeholders (e.g., patients and their family, health care providers, broader health system, and local community or entire society such as a country), and can demonstrate change required to indicate success of virtual care.²⁴ Altogether, the evaluation guidance documents recommended performing evaluations with the following outcome categories and examples of related measurements:

• Effectiveness and quality of clinical care including safety outcomes

  • Frequency of positive (e.g., successful virtual consultations) or negative events (e.g., adverse events)

  • Measures of independence, mortality, functional status, and disease morbidity

  • Patients’ knowledge and understanding of treatment plan

• Time and travel

  • Wait times (to initiation of appointment from check-in)

  • Time on wait list

  • Travel time and distance to appointment

  • Number of free hours

  • Time saved (e.g., more time for personal priorities such as work or family responsibilities)

• Financial and operational impact

  • Total cost of medical care

  • Cost per episode of care

  • Cost savings

  • Virtual care program expenses (e.g., IT infrastructure, technology set-up, and maintenance costs)

• Participation

  • Attendance

  • No-show rate

  • Number of patients treated

• Health care utilization

  • Duration and frequency of appointments

  • Need for follow-up appointments (particularly in-person follow-up that could not be addressed virtually)

  • Hospital admissions

  • Emergency department visits

• Technology experience including feasibility

  • Ease of using technology

  • Ability of providers to conduct required assessments and treatments

  • Percentage of sessions that involve technical difficulties affecting session quality and ability to provide services

  • User convenience (patient or provider)

  • Comfort with using virtual care applications and processes

  • Technical issues

  • Stability and reliability of the virtual care system and method used to transmit information

• User satisfaction

  • Satisfaction surveys of patients and providers

  • Provider rating of virtual care and willingness to expand virtual care in their practice

  • Patient rating of quality of virtual care sessions and willingness to participate in future sessions

• Barriers and facilitators or measures of health equity

  • Percentage of patients who delay care due to access barriers (e.g., lack of access to broadband or technology)

  • Percentage of patients who are able to receive virtual care in their desired language

  • Percentage of patients with disabilities who are able to receive care virtually through adaptive technologies

  • Access to health services for those living in rural and urban communities and/or medically underserved areas

Altogether, the following considerations were generally highlighted including societal aspects (e.g., economic, legal, political, and sociocultural/socioeconomic considerations); type of care (e.g., primary health care, hospital services, rehabilitation); barriers (e.g., access to technology, feasibility of implementation, financial or time costs); stakeholders (e.g., acceptability and perception of patients and their family and caregivers, associations of health professionals, regulators or policy decision-makers, community, and academic and research groups); and measurement aspects (e.g., accessibility, effectiveness, efficiency, technology usability, and patient or provider experience).

Queensland Health specified that the following key decisions and considerations should be integrated into the planning of a virtual care evaluation.²⁴

1. Identifying the scope of the virtual care service to be evaluated ― the scope of service needs to allow for a comprehensive evaluation yet allow for timely, feasible, non-complex, and non‒resource-intensive implementation

2. Identifying the appropriate comparator ― the comparison needs to be relevant and comparable in real-world clinical practice, with consideration of differences in comparator groups (e.g., for time comparisons, ensure there are no major additional changes that occurred between the time periods)

3. Selecting the appropriate timing of the evaluation – ensure the evaluation is not skewed by pilot data or early stages when providers and patients are adjusting to virtual care, it is not influenced by conditions of particular data collection periods such as seasonal demands that influence staff resourcing, and strategic or organizational factors (e.g., funding)

4. Selecting the appropriate duration of the evaluation ― ensure the evaluation period facilitates the collection of a sufficient amount of data required for meaningful findings, appropriate representation of real-world clinical practice (longer data collection periods are less likely to be influenced by timing), and allow for an adequate measure of outcomes (time frames required to measure the outcome appropriately and comprehensively).²⁴

The Pan American Health Organization²³ and Heart and Stroke Foundation of Canada^21,22 noted that evaluations of virtual care should assess the individual virtual care sessions, as well as the virtual care program as a whole. Various evaluation guidance documents noted that evaluations should be systematic, performed regularly, and reflect the stage of implementation of virtual care to adequately reflect real-world conditions.^21-23 Namely, the Pan American Health Organization proposed indicators based on chronological impact ― short-term (e.g., increase in virtual consultations conducted), medium-term (e.g., hospitals participating in virtual care in reference to the national total), and long-term (e.g., average savings over the previous year). Additionally, it was noted that during early phases virtual care implementation, evaluations should focus on attaining the initial objectives and on how strategies are aligned with the goals or targets. During mid-stages of implementation, evaluations should focus on optimization; and during established phases, evaluations should focus on social and economic benefits.²³ Similarly, Queensland Health specified pivotal factors for the following stages of virtual care evaluation:

• Pre-introduction stage ― technical and financial feasibility, service need, preparedness of staff and patients, and legal or ethical issues

• Pilot or trial stage ― clinical effectiveness, legal or ethical issues, technical feasibility, service process (administrative or operational), safety and quality

• Adoption or implementation ― clinical effectiveness, health outcomes, service efficiency, safety and quality, and acceptability and satisfaction

• Established or translational stage ― health outcomes, acceptability and adoption, access, efficiency, and technology (scalability).²⁴

The Pan American Health Organization²³ and Queensland Health²⁴ highlighted that costs or savings are not limited to monetary or financial measures (e.g., out-of-pocket travel expenses) but can also be represented with time. For example, the time needed for patients to travel, wait, and attend the appointment, and staff to travel and conduct the appointment, takes time away from personal priorities (e.g., work [specifically for patients and caregivers] and family commitments such as childcare). Additionally, specific information related to the evaluation of economic outcomes were highlighted by the Pan American Health Organization²³ and Queensland Health.²⁴ The Pan American Health Organization specified that indicators informed by the framework of health economics should be considered including cost-benefit and cost-utility estimates such as out-of-pocket expenditures and evaluation of price and quality. Queensland Health highlighted that perspective is very important for evaluations that have an economic component (i.e., analyses of outputs in relation to inputs). Perspective determines which inputs (costs) and outputs (benefits or effects) are included in the analysis and how the outcome is interpreted.²⁴

Summary of Included Evaluations of Virtual Care

The identified evaluations assessed the following measures of value: economics, health care utilization, participation, clinical outcomes, and appropriateness of prescribing (for evidence specific to prescriptions). Economic outcomes included cost in the context of patients (e.g., parking and travel costs) or provider expenses, and as specific measures such as cost efficiency, service cost per patient, cost per episode, and total service cost through cost analyses. Health care utilization outcomes included number and duration of medical appointments; use of medical resources (e.g., lab tests); and need for follow-up appointments (e.g., related to the same health concern), hospital admissions, and emergency department visits. Of note, findings related to the number of virtual care appointments versus number of in-person appointments were only included in this ES when there was not a COVID-19 timeline focus or comparison in the study (i.e., informed by the SRs), as the use of virtual care increased considerably during the pandemic, while availability of in-person care was reduced. Participation outcomes included attendance or show rates, non-attendance or no-show rates, cancellations, completion rate, and adherence to and discontinuation of a series of medical appointments. Participation outcomes were assessed as various measures such as rate and count (e.g., number of sessions attended). Clinical outcomes were specific to the disease or health context; for example, Frank et al. (2021) assessed clinical outcomes through clinician-rated scales such as the Clinical Global Impressions scales ― specifically measures of severity and improvement ― as the study was conducted in a primary care clinic with integrated mental health services.¹⁶ The SR performed by Carrillo de Albornoz et al. (2021) included studies that assessed a variety of clinical outcomes specific to the disease or health context as rate, count, or score changes.¹² Namely, continuous abstinence rate, nicotine dependence, and cravings were assessed in evaluations of smoking cessation management; body composition (height and weight) changes in evaluations of obesity; readmissions to rehabilitation in evaluations of malnutrition; parenting self-efficacy (self-reported) and session frequency in evaluations of management of individuals with disability; hemoglobin A1C (HbA1C) levels in evaluations of type 2 diabetes; and infant feeding and maternal satisfaction with care in evaluations of postpartum care. In evaluations of mental health treatment, acceptance or satisfaction of use (i.e., positive attitude toward virtual care) and adherence to treatment, symptom severity, and pain intensity were measured outcomes, in addition to a variety of clinical scales such as the Hamilton Depression Rating Scale (HAM-D), WHO Disability Assessment Schedule (WHODAS), Geriatric Depression Scale (GDS), Beck’s Depression Inventory (BDI), Penn State Worry Questionnaire (PSWQ), and the 36-Item Short Form survey (patient satisfaction and quality of life). Appropriateness of prescribing was evaluated through outcomes of guideline-concordant drug management or guideline-concordant prescription rates according to local or national guidelines and compliance to the recommendation of clinical scale scores to guide prescribing. The latter was assessed by Li et al. (2021) and the clinical scale scores used to guide prescribing were Centor or FeverPAIN for antibiotic prescription in individuals with acute tonsilitis.¹⁰

Carrillo de Albornoz et al. (2021) highlighted that virtual care may not be suitable for all patients. In-person appointments may be preferred by patients with high-risk conditions who require physical examinations or who cannot communicate adequately by telephone or video (i.e., not adept with or lacking technology). Alternatively, patients most likely to benefit from virtual consultations may be those with chronic conditions who require many follow-up appointments and patients with difficulties travelling to the health centre (e.g., work reasons, geographical limitations, and physical disability). Carrillo de Albornoz et al. (2021) reported that discontinuation rates reported in 1 included SR varied from 0% to 72%; examples of various types of care eliciting lower discontinuation rates using telephone health services included cognitive behavioural therapy patients in primary care, cancer patients, and low-income adults receiving primary care.¹² These examples highlight that virtual care may be more effective in certain health contexts.¹² Identified evaluations suggested that virtual care may be more cost-effective and reduces the cost per episode and patient expenses (e.g., travel and parking costs), as virtual care reduces the duration of appointments per patient. However, virtual care may increase the total number of individuals treated, which would increase overall health care spending and utilization. Additionally, it is unclear if virtual care reduces the use of medical resources and the need for follow-up appointments, hospital admissions, and emergency department visits. Overall, identified evaluations of virtual care assessing participation outcomes, clinical outcomes in various health contexts, and appropriateness of prescribing reported variable findings when compared to in-person care. Namely, some evidence reported that virtual care reduced attendance (e.g., reduced attendance rates) or improved attendance (e.g., increased completion rate and decreased cancellations and no-show rates). Some evidence suggested that virtual care improved clinical outcomes (e.g., in primary care with integrated mental health services, symptom severity decreased and symptoms improved) or had a similar effect on clinical outcomes (e.g., in patients with depression and alcohol abuse), and some studies noted that virtual care improves appropriateness by increasing guideline-based or guideline-concordant antibiotic management or elicits no difference compared in-person care.

Limitations

This ES may not provide an entirely comprehensive review of virtual care evaluations in primary care across Canada and internationally, as the literature search used to inform the findings were limited ― namely, to English-language documents published between January 1, 2019 and September 1, 2021. Accordingly, the included evaluations were mostly performed during the first wave of the COVID-19 pandemic when there were no vaccines available and global lockdowns were enforced. Although this pandemic served as an opportune time to evaluate virtual care in primary care due to the timely demand for virtual care, this pandemic introduced distinct factors that limit the generalizability of the included evidence to inform virtual care evaluations or implementation outside the context of COVID-19; for example, the lack of preparedness by health care providers to rapidly switch to predominantly using virtual care health services and greater limitations with access to internet or phones among patients due to the closure of public spaces such as libraries. The latter disproportionately affected those without personal phones, computers, or access to good-quality technology (e.g., individuals of lower socioeconomic status). This is supported by the Queensland evaluation guide that specified that pre- and post-implementation comparisons are most appropriate when there are no major changes between measurement periods.²⁴ Namely, in-person versus virtual care comparisons using pre‒COVID-19 and COVID-19 periods could be influenced by particular factors such as specific needs for health care (e.g., COVID-19 health concerns), poorer mental health of the overall population, higher stress levels, increased competing priorities or responsibilities (e.g., assisting children with virtual school due to lockdown measures), and reduced assistance particularly for elderly and/or chronically ill patients (e.g., restrictions on caregivers or health care support workers providing assistance with the virtual care appointment due to lockdowns and need for physical distancing).

Additionally, the time-based comparison (e.g., pre‒COVID-19 versus COVID-19 periods) likely involves comparisons of different groups of participants; thus, the samples representing in-person and virtual care are different. Accordingly, the samples may not be comparable in characteristics such as demographics (e.g., age, sex, socioeconomic status), lifestyle, medical history, diagnosis, and severity of symptoms or stage of disease, even among those with the same diagnosis, which introduces confounding and may bias findings. For example, the comparison of the number of emergency department visits between virtual and in-person care may be due to 1 group having a greater incidence of diagnoses that increase the likelihood of an emergency department visit. Further, the supporting evidence consisted of individual studies or SRs evaluating various services within primary care and outcomes were reported per person or as a total. This complicates the interpretations and limits the generalizability of the findings because of different health care services being associated with particular considerations. For example, some indications for a health consultation require 1 appointment (e.g., antibiotic prescription) versus others that may be more chronic or severe such as mental health indications, which require a series of appointments for adequate treatment. This variability in the need for the number of appointments may influence participation outcomes. Additionally, for health concerns that require a series of appointments, adherence becomes relevant and may skew measures such as the number of appointments attended, as the overall number may increase due to the reason of the visit and not due to the use of a virtual or in-person modality. Overall, it is unclear whether the findings are due to the effect of virtual care versus in-person care or whether they were due to the characteristics of the analyzed samples of the included virtual care evaluations and evidence that informed the included evaluation guidance documents.

It should also be highlighted that, among the included evidence, socioeconomic and cultural considerations were discussed in a broad manner. Socioeconomic and cultural considerations can be sub-categorized and represented as vulnerable populations. Some of these considerations may be of greater relevance in particular jurisdictions and among specific populations. Vulnerable populations ― which may include racial and ethnic minorities, people who have an economic disadvantage (e.g., those who are uninsured and/or have a low-income), women or other gender identities (e.g., non-binary), people with unstable or without housing, individuals with stigmatized health conditions (e.g., HIV, severe mental illness), individuals with chronic health conditions, and older adults ― experience more barriers in accessing health care.²⁷ Health and barriers to health care are further influenced by social factors and level of education.²⁷

The included evidence did not discuss approaches to evaluating virtual care in primary care for Indigenous peoples in Canada or other countries. Indigenous peoples and Indigenous communities in Canada face barriers to equitable access to health care.^28,29 The lack of literature identified in this ES pertaining to Indigenous peoples is reflective of a broader evidence gap in primary research, which has been highlighted in other literature, as well.²⁸ For many Indigenous peoples, access to health care is complicated by the historical and ongoing effects of colonialism including the isolated and/or remote location of Indigenous communities, segregation, racism, improper treatment, and a lack of cultural competency on the part of some health care providers. Accordingly, Indigenous peoples are more likely to be economically disadvantaged, have chronic diseases, and experience homelessness; thus, Indigenous peoples are a relevant vulnerable population in Canada whose health care needs should be properly addressed.^28,29

Access to health care for Indigenous populations in Canada is a substantial issue and access to virtual care may be considerably limited, as many will not have access to technology, particularly that of high quality, such as internet connectivity or devices to facilitate virtual care.^28,29 Further research is needed to ensure that virtual care meets the needs of Indigenous peoples and Indigenous communities, which may be different from other vulnerable groups in Canada.^28,30

None of the evaluations of virtual care were conducted in Canada and the evaluation guidance documents only have representation of Canada through 2 publications from the Heart and Stroke Foundation of Canada.^21,22 The document published by the Pan American Health Organization was not informed by Canadian representatives. Additionally, economic evidence on virtual care evaluations was limited, as this was informed by 2 included SRs.^12,13 Economic outcomes are pivotal to evaluations of virtual care, as all included evaluation guidance documents noted that measures of cost or savings should be performed to assess the financial and operational impact.^20-26

Further, some evaluations only included audiovisual (video) modalities and others combined audio only (e.g., telephone) and video to implement virtual care. Combining different types of virtual care does not provide informative evidence, as audio versus video appointments have their own technical considerations, benefits, and limitations. There was also a mix of age included in the evaluations (i.e., adults only, pediatric only, or both adults and pediatric-aged individuals). Age is a confounder for health outcomes and the upper and lower spectrums of age may be associated with variable access or ability to independently use technology to facilitate virtual care. One of the included studies found that age was associated with an increased risk of hospitalization if the initial visit was a virtual care visit versus an in-person visit.¹⁸ Overall, the included evidence is limited in its generalizability because of considerable differences within and between treatment (e.g., virtual care versus in-person care) groups and is limited in the Canadian context. The quality of included evidence is low and findings may be associated with a high risk of bias, as 8 of the 11 evaluations were non-randomized comparative studies. Of note, a formal critical appraisal was not performed for this ES.

Conclusions and Implications for Decision- or Policy-Making

This ES was informed by a literature search to present a broad overview of recently published literature (between January 1, 2019 and September 1, 2021) and included 11 virtual care evaluations in primary care and 7 evaluation guidance documents of virtual care evaluation across various countries, including Canada and internationally (e.g., US, UK, and Australia).

The included evidence noted that evaluations of virtual care should consider economic, legal, political, and sociocultural factors; relevance and priorities of stakeholders (e.g., patients and their family and caregivers, associations of health professionals, regulators or policy decision-makers, community, and academic and research groups); and the timeliness, feasibility, and complexity of measuring relevant outcomes (e.g., effectiveness and quality of clinical care including safety outcomes, time and travel, financial and operational impact, participation outcomes, health care utilization, technology experience including feasibility, user satisfaction, and barriers and facilitators or measures of health equity). Notably, evaluations should be specific to the type of health care or service and disease or health context. The included evaluation guidance documents also noted that evaluations should be systematic, performed regularly, and reflect the stage of implementation of virtual care to adequately reflect real-world conditions, as there are specific considerations associated with each stage. Additionally, the evaluation guidance documents noted that evaluations should assess the individual virtual care sessions, as well as virtual care sessions overall. Findings from presently available virtual care evaluations lack certainty, are quite limited in generalizability, particularly in the Canadian context, and are generally inconclusive because of the variability in the results. Virtual care may be more cost-effective and reduce the cost per episode and patient expenses but may increase overall health care spending and health care utilization through an increase in the total number of appointments. This may be because virtual care appointments are shorter in duration, which allows for more appointments in a given time. Additionally, the effect of virtual care on participation outcomes such as attendance, clinical outcomes such as the change in clinical scores, and appropriateness of prescribing is unclear. Nevertheless, the evidence suggests that virtual care may not be suitable for all patients. In-person appointments may be preferred by patients with high-risk conditions who require physical examinations or who cannot communicate adequately by telephone or video, whereas virtual care benefits those with chronic conditions who require many follow-ups and patients with travel limitations (e.g., work reasons, geographical limitations, and physical disability). For example, there is a considerable challenge in deciding whether virtual care should be implemented for older adult populations because of the complex integration of health and social needs. Although decision tools were out of scope for this ES, such guidance exists; for example, Provincial Geriatrics Leadership Ontario established a virtual care task group to devise a decision tool kit.³¹ Nevertheless, other decision tools used by different jurisdictions may exist.

Further, the economic-based evidence was limited. Therefore, well-designed economic evaluations of virtual care are needed and may help inform the implementation of virtual care and support evaluations of virtual care in Canada ― namely, that cost and feasibility considerations may be specific to the patient, health care provider, or health care system as a whole. Therefore, selection of the perspective of economic evaluations is pertinent, as it influences the inputs (costs), outputs (benefits or effects), and interpretation of outcomes. The included guidance also noted that the selection of the appropriate perspective should consider other allied health teams and services. Further, public versus private health care settings should be differentiated, as these are associated with differences in accessibility to technology of varying quality (e.g., speed of internet or performance level of technology) and in the amount and source of coverage (for the patient) or funding (for the provider). The included guidance also highlighted that costs are not limited to out-of-pocket expenses but can also include the impact on time measures (e.g., time spent travelling and the need to take time off work). Moreover, the included evidence did not allude to provincial billing or fee structures in Canada. A previous CADTH Horizon Scan noted that all provinces and territories except for Nunavut (at the time of evidence synthesis in June of 2021) implemented fee codes for virtual visits during the COVID-19 pandemic.³² However, guidance from the Canadian government varied regarding billing processes.³² Therefore, future research of evaluations of virtual care should consider the effectiveness and feasibility of billing processes. For example, as Canadian jurisdictions rapidly adopted billing processes, assessments should be performed to gauge cost appropriateness and to gauge if selected billing costs are sustainable or require changes in health care funding. Furthermore, virtual care can enable care across borders, which may improve access but complicate reimbursement or introduce additional challenges. A CADTH policy brief on interjurisdictional medical licensing to support telemedicine may help inform this.³³

Overall, there is a need for higher-quality virtual care evaluations that consist of comparison groups that are more similar in characteristics such as age, medical history (e.g., diagnosis and severity of symptoms), and socioeconomic status, as these are considerable confounders. Research specific to vulnerable populations is needed. In countries, such as Canada, there is a considerable need for evidence specific to the approaches of evaluations of virtual care primarily used by Indigenous individuals to ensure they receive adequate and appropriate care. This lack of evidence specific to Indigenous peoples is a considerable limitation and this evidence gap emphasizes the need for decision-makers to ensure that the voices of people who are often excluded or marginalized are included when designing, implementing, and evaluating virtual care in primary care. Additionally, there is a need for evaluations and guidance that report findings separately for various health care types and services (e.g., separating hospital and private practice data) and virtual care modalities (e.g., separating audiovisual [video] versus audio only). Further, there is a need for evaluations of virtual care conducted during periods that are not restricted to the early waves (e.g., first wave) of the COVID-19 pandemic to minimize confounding. As this pandemic is currently ongoing and at different stages around the world (e.g., some countries have access to a greater supply of vaccines and/or higher rates of vaccinated individuals), continuous longer-term research that incorporates transition periods from various countries is needed, especially as research on virtual care before this pandemic is limited considering that virtual care was not commonly implemented in the same manner. As further research is conducted, it may be appropriate for updates to this ES to also be conducted. Altogether, additional evaluations with the aforementioned characteristics are needed for more relevant informative comparisons with in-person care to facilitate decision- and policy-making regarding evaluations of virtual care in primary care in Canada.

References

1.Canadian Institute for Health Information. How COVID-19 affected physician services. 2021. Accessed 2021 Aug 17.

2.Canadian Institute for Health Information. COVID-19’s impact on physician services. 2021; https://www.cihi.ca/en/covid-19-resources/impact-of-covid-19-on-canadas-health-care-systems/physician-services. Accessed 2021 Dec 14.

3.Virtual care recommendations for scaling up virtual medical services: report of the Virtual Care Task Force. Ottawa (ON): Canadian Medical Association, College of Family Physicians of Canada, Royal College of Physicians and Surgeons of Canada; 2020: https://www.cma.ca/sites/default/files/pdf/virtual-care/ReportoftheVirtualCareTaskForce.pdf. Accessed 2021 Aug 18.

4.Shaw J, Jamieson T, Agarwal P, Griffin B, Wong I, Bhatia RS. Virtual care policy recommendations for patient-centred primary care: findings of a consensus policy dialogue using a nominal group technique. J Telemed Telecare. 2018;24(9):608-615. PubMed

5.Adopting and integrating virtual visits into care: draft clinical guidance for health care providers in Ontario. Toronto (ON): Health Quality Ontario; 2020: https://quorum.hqontario.ca/Portals/0/Users/170/54/10154/Draft%20Clinical%20Guidance_Adopting%20and%20integrating%20virtual%20visits%20into%20care_V1.pdf?ver=2020-03-13-091936-370. Accessed 2021 Aug 18.

6.Analysis of the current and potential benefits of virtual care in Canada. Toronto (ON): Canada Health Infoway; 2020: https://www.infoway-inforoute.ca/en/component/edocman/3819-analysis-of-the-current-and-potential-benefits-of-virtual-care-in-canada/view-document?Itemid=101. Accessed 2021 Aug 18.

7.Peckham A, Ho J, Marchildon G. Policy innovations in primary care across Canada. Toronto (ON): North American Observatory on Health Systems and Policies; 2018: https://ihpme.utoronto.ca/wp-content/uploads/2018/04/NAO-Rapid-Review-1_EN.pdf. Accessed 2021 Oct 9.

8.O'Gurek DT. Designing and evaluating COVID-19 protocols for an office-based opioid treatment program in an urban underserved setting. J Am Board Fam Med. 2021;34(Suppl):S136-S140. PubMed

9.Shaffer AD, Dohar JE. Evidence-based telehealth clinical pathway for pediatric tympanostomy tube otorrhea. Int J Pediatr Otorhinolaryngol. 2020;134:110027. PubMed

10.Li C, Ong C, Morris A, Woollons I, Ashfaq A, Jagatia R. Evaluating the appropriateness of antibiotic treatment of tonsillitis during COVID-19 in the North Wale primary healthcare setting. J Prim Care Community Health. 2021;12:21501327211003687. PubMed

11.Han SM, Greenfield G, Majeed A, Hayhoe B. Impact of remote consultations on antibiotic prescribing in primary health care: systematic review. J Med Internet Res. 2020;22(11):e23482. PubMed

12.Carrillo de Albornoz S, Sia KL, Harris A. The effectiveness of teleconsultations in primary care: systematic review. Fam Pract. 2021;19:19. PubMed

13.Nguyen OT, Alishahi Tabriz A, Huo J, Hanna K, Shea CM, Turner K. Impact of asynchronous electronic communication-based visits on clinical outcomes and health care delivery: systematic review. J Med Internet Res. 2021;23(5):e27531. PubMed

14.Chakawa A, Belzer LT, Perez-Crawford T, Yeh HW. COVID-19, telehealth, and pediatric integrated primary care: disparities in service use. J Pediatr Psychol. 2021;46(9):1063-1075. PubMed

15.Drerup B, Espenschied J, Wiedemer J, Hamilton L. Reduced no-show rates and sustained patient satisfaction of telehealth during the COVID-19 pandemic. Telemed J E Health. 2021;27(12):1409-1415. PubMed

16.Frank HE, Grumbach NM, Conrad SM, Wheeler J, Wolff J. Mental health services in primary care: Evidence for the feasibility of telehealth during the COVID-19 pandemic. J Affect Disord Rep. 2021;5:100146. PubMed

17.Gmunder KN, Ruiz JW, Franceschi D, Suarez MM. Factors to effective telemedicine visits during the COVID-19 pandemic: cohort study. JMIR Med Inform. 2021;9(8):e27977. PubMed

18.Phillips JC, Lord RW, Davis SW, Burton AA, Kirk JK. Comparing telehealth to traditional office visits for patient management in the COVID-19 pandemic: A cross-sectional study in a respiratory assessment clinic. J Telemed Telecare. 2021:1357633X21990197.

19.Ray KN, Martin JM, Wolfson D, et al. Antibiotic prescribing for acute respiratory tract infections during telemedicine visits within a pediatric primary care network. Acad Pediatr. 2021;21(7):1239-1243. PubMed

20.UCL Partners. 10. How do we evaluate our non face-to-face clinics? 2021; https://uclpartners.com/non-face-to-face-clinics-resource/capturing-impact-and-measuring-success/. Accessed 2021 Oct 14.

21.Virtual healthcare (telestroke) implementation toolkit. Ottawa (ON): Heart and Stroke Foundation; 2020: https://www.heartandstroke.ca/-/media/1-stroke-best-practices/csbpr7-virtualcaretools-13may2020. Accessed 2021 Oct 14.

22.Virtual cardiovascular prevention and rehabilitation implementation toolkit. Ottawa (ON): Heart and Stroke Foundation; 2021: https://www.heartandstroke.ca/-/media/1-stroke-best-practices/vcr-toolkit-final-2021.ashx?rev=e2d73b476e6e4ef1abc09624992566d0. Accessed 2021 Oct 14.

23.Defining evaluation indicators for telemedicine as a tool for reducing health inequities. Washington (DC): Pan American Health Organization; 2016: https://iris.paho.org/bitstream/handle/10665.2/28562/PAHOKBR16006-eng.pdf?sequence=8&isAllowed=y. Accessed 2021 Oct 14.

24.Evaluation resource guide: allied health telehealth capacity building project. Brisbane (AU): Queensland Health; 2016: https://www.health.qld.gov.au/__data/assets/pdf_file/0024/451077/telehealth-evalguide.pdf Accessed 2021 Oct 14.

25.Return on health: moving beyond dollars and cents in realizing the value of virtual care. Chicago (IL): American Medical Association; 2021: https://www.ama-assn.org/system/files/2021-05/ama-return-on-health-report-may-2021.pdf. Accessed 2021 Oct 14.

26.Creating a framework to support measure development for telehealth. Washington (DC): National Quality Forum; 2017: https://www.qualityforum.org/publications/2017/08/creating_a_framework_to_support_measure_development_for_telehealth.aspx. Accessed 2021 Oct 14.

27.Vulnerable populations: who are they? Am J Manag Care. 2006;12(13 Suppl):S348-352. PubMed

28.Dawson AZ, Walker RJ, Campbell JA, Davidson TM, Egede LE. Telehealth and indigenous populations around the world: a systematic review on current modalities for physical and mental health. Mhealth. 2020;6:30-30. PubMed

29.Access to health services as a social determinant of First Nations, Inuit and Métis health. Prince George (BC): National Collaborating Centre for Indigenous Health; 2019: https://www.nccih.ca/docs/determinants/FS-AccessHealthServicesSDOH-2019-EN.pdf. Accessed 2021 Dec 15.

30.Camp P, Girt M, Wells A, et al. Virtual care for Indigenous populations in Canada, the United States, Australia, and New Zealand: protocol for a scoping review. JMIR Res Protoc. 2020;9(12):e21860. PubMed

31.Leveraging virtual care strategies in delivery of Comprehensive Geriatric Assessment (CGA): A decision tool kit. [Cobourg (ON)]: Provincial Geriatrics Leadership Ontario; 2021: https://rgps.on.ca/wp-content/uploads/2021/02/2021-Mar-2-Final-Report-Tool-Kit-Formatted.pdf. Accessed 2021 Dec 14.

32.Gray C, Mason J, Loshak H. An overview of direct-to-patient virtual visits in Canada. Can J Health Technol. 2021;1(6). https://www.cadth.ca/sites/default/files/hs-eh/EH0091%20Virtual%20Visits%20Final.pdf. Accessed 2021 Dec 14.

33.Inter-jurisdictional medical licensing to support telemedicine. (CADTH Policy brief). Ottawa (ON): CADTH; 2020: https://www.cadth.ca/sites/default/files/hta-he/cadth-policy-brief-interjurisdictional-licensing-final.pdf. Accessed 2021 Dec 15.