Providing valid evidence for decision-making: the Drug Safety and Effectiveness Network Methods and Applications Group in Indirect Comparisons (DSEN MAGIC)

The collaborating centres are pan-Canadian networks of researchers that were launched via open calls for team grants focused in various methodologic areas such as observational studies, pharmacogenetics studies, and network meta-analysis (NMA), which compares multiple treatments simultaneously in a single analysis through combining direct and indirect evidence from primary studies (Caldwell et al. 2005). They were created to organize researchers with relevant, internationally recognized methods experts who could respond efficiently to queries from knowledge users (e.g., policy makers) and engage with these knowledge users to enhance research dissemination and capacity building. Moreover, these collaborating centres were mandated to build capacity in Canada to create future generations of researchers with expertise in drug safety and effectiveness.

In 2011, an open call for a collaborating centre in NMA was launched and three multidisciplinary, pan-Canadian research teams were funded for three years to respond to relevant research questions from decision-makers (called queries). While some base support for each team was provided for demonstration projects, individual grant applications were prepared and submitted for peer review for each new query from knowledge users. During this 3-year period, more than 28 queries were responded to and 15 knowledge syntheses (comprehensive literature search using reproducible methods, (CIHR 2010)) were conducted, leading to 25 peer-reviewed publications in high impact journals, including BMJ (British Medical Journal) and CMAJ (Canadian Medical Association Journal), amongst others. More than 50 presentations and courses were delivered by the research teams for various knowledge users. Two investigators (BH and ACT) involved with the teams received CIHR New Investigator Awards in drug safety and effectiveness; 15 trainees were involved in the teams’ activities. Based on the success, productivity, and established collaborations across the teams, in 2014, the three teams were invited to submit a single application for renewal. This application led to the creation of MAGIC: Methods and Application Group for Indirect Comparisons. MAGIC was funded $13,128,862 for 9 years overall (originally from 2014 to 2019 and renewed from 2019 to 2022).

MAGIC was created to improve pharmacovigilance and post-market surveillance in Canada by continuing to provide high-quality knowledge synthesis and an environment of research excellence. Learnings from the initial 3 years identified the need for more nimbleness to respond to queries; as such, the base collaborating centre funding was increased to remove the barrier of applying for peer-reviewed funding for each query received from the knowledge users. This change was in direct response to knowledge user feedback to have a more efficient response to the queries. Similarly, the review teams identified through engagement with knowledge users that queries often required access to different knowledge synthesis methods outside of NMA including rapid reviews, scoping reviews, and overviews of reviews amongst others. This review expertise was available through MAGIC.

The MAGIC team includes 42 members from six provinces and four countries and represents an international collaboration of researchers with recognized methodologic expertise to address questions of drug safety and effectiveness and cost effectiveness. Members’ expertise includes different research disciplines such as knowledge synthesis, biostatistics, knowledge translation (KT), pharmacoepidemiology, pharmacoeconomics, sex and gender analysis, and health technology assessment. Clinical specialties include nursing, medicine, pharmacy, and clinical pharmacology amongst others. Of note, team members are internationally recognized for their works to advance methods in NMA (e.g., PRISMA NMA (Hutton et al. 2015), assessing inconsistency (Veroniki et al. 2013), including observational studies (Cameron et al. 2015), risk of bias assessment (Brown et al. 2014; Cameron et al. 2015), rapid reviews (e.g., handbook,(Tricco et al. 2017; Tricco et al. 2015)), scoping reviews (e.g., PRISMA ScR (Tricco et al. 2016a; Tricco et al. 2018)) and various other knowledge synthesis methods (Tricco et al. 2016b) as well as KT (Graham et al. 2006; Straus et al. 2011b; Strifler et al. 2018). Most importantly, MAGIC includes knowledge users and operates via an integrated KT approach whereby knowledge users are engaged from identification of the question through to research design, completion (Straus et al. 2013), and dissemination.

Knowledge users (including Health Canada, provincial partners, CADTH) can submit queries regarding drug safety and effectiveness to the DSEN Coordinating Office using a standardized template. A DSEN Scientific Advisory Committee that includes representation from the CIHR, Health Canada, CADTH, CIHI, and the collaborating centres discusses query feasibility. Each collaborating centre conducts a feasibility assessment from their team’s methodologic perspective; for example, some queries may best be answered by observational studies and others through a systematic review and NMA, a prospective observational study, or a combination of methods. The DSEN Coordinating Office then asks the collaborating centres to present their feasibility assessments at a monthly Scientific Advisory Committee meeting with the query requestor (i.e., knowledge user) in attendance. After this meeting, the requestor considers which of the approaches they feel would best meet their needs. The CIHR Coordinating Office then informs the relevant collaborating centres. The coordination of this process via the CIHR Coordinating Office typically takes 1–3 months.

MAGIC uses a five-phase process to answer queries (Figure 1), which has been iteratively developed over the past 10 years. The cross-cutting principles to our approach focus on ensuring knowledge users’ needs for rigorous knowledge syntheses are met in a timely fashion, building capacity and mentorship, demonstrating research excellence, and using an integrated KT approach. In Phase 1, the DSEN coordinating office informs MAGIC about the decision from the query requestor; MAGIC immediately asks to meet with the requestor and a teleconference is organized. In Phase 2, MAGIC meets with the requestor to clarify their needs to inform the query (outlining the patient, interventions, comparators, outcomes, concept, and context). During this meeting, query restrictions are outlined (such as age, language) and suitable methods for knowledge user engagement are identified as well as desired format of final deliverables (e.g., report, slide deck). In Phase 3, a work plan is created using the template specified by the DSEN Coordinating Office; it is sent to the knowledge user and the Coordinating Office for Review. The work plan includes the methods, budget, KT strategy, project update frequency, and timelines. MAGIC offers to present and discuss the work plan with the knowledge user. MAGIC conducts the research during Phase 4 and provides regular updates via the DSEN Coordinating Centre. Preliminary reports with the knowledge users are provided and knowledge exchange is offered via a webinar to discuss results. Tailored end-of-grant KT strategies are implemented in Phase 5. For example, MAGIC incorporates all feedback from the knowledge users, creates a final report, offers a webinar to present results and submits a publication to an open access, peer-reviewed journal. Knowledge users are invited co-authors on all publications. Publications are shared with the DSEN coordinating office. A one-page research brief is posted on the MAGIC website and disseminated via social media (e.g., Twitter, LinkedIn).

Queries may be declined for a variety of reasons. For example, a knowledge synthesis may not be feasible for the query as there may not be any primary studies on the topic (e.g., such as for a new drug that has not yet been tested in a randomized trial or no post-marketing surveillance data are yet available). If a recent, high-quality knowledge synthesis has already been completed, there is no need for another synthesis, thereby avoiding duplication of effort and research waste. Another common reason for declining a query is that the question is best answered by another study design such as a prospective observational study.

Once a query is approved, MAGIC uses a standardized nine-step method for knowledge synthesis as outlined in Figure 2. We use standardized methods to reduce redundancy, promote independent reproducibility, increase efficiency, and ensure best practices in knowledge synthesis conduct such as methods in the Cochrane Handbook for Systematic Reviews (Higgins et al. 2021), JBI for Scoping Reviews (Peters et al. 2020), and JBI Methods Guide for Synthesizing Qualitative Evidence (Aromataris 2020). All steps are done in an iterative fashion with input from the knowledge users throughout. In Step 1, we match the query to the relevant knowledge synthesis method, using a freely available tool we created (KTP 2019). Question formulation and refinement occur in Step 2 and sex and gender are considered in all questions as appropriate. The protocol and work plan are developed in Step 3. The protocol follows relevant reporting guidelines (e.g., PRISMA-P (Shamseer et al. 2015)) and with permission from the knowledge user, a one-page protocol brief is posted on the MAGIC website. The protocol is peer-reviewed and registered (e.g., with PROSPERO (2021) for systematic reviews and Open Science Framework (OSF 2021) for other reviews). The literature search is conducted in Step 4, by experienced librarians using multiple electronic databases and grey literature (CADTH 2011). Online software (e.g., DistillerSR (2021), Synthesi.SR (Synthesi 2014)) is used to manage and standardize screening, including titles and abstracts and full text (Step 5). Data abstraction occurs in Step 6 and we assess the evidence quality in Step 7 using GRADEpro (2021) software and relevant risk of bias (Sterne et al. 2019) tools. In Step 8, we complete the synthesis, which includes a descriptive summary of the studies. If more than one included study reports an outcome specified in the research query, and where appropriate and feasible, we conduct a meta-analysis. NMA is considered in partnership with knowledge users when there are multiple interventions and the knowledge users want to make inferences regarding their relative effectiveness. Reporting the knowledge synthesis is Step 9 and involves use of relevant reporting guidelines from the EQUATOR Network to ensure transparency.

One of MAGIC’s goals is to provide a positive, nurturing, and stimulating environment for trainees interested in knowledge synthesis including NMA. We use the successful Strategic Training Initiative in Health Research (STIHR (Straus et al. 2011a)) training framework, developed over 10 years ago for KT Canada. We leverage other CIHR-funded initiatives including Drug Safety and Effectiveness Cross-disciplinary Training (DSECT) Program and the Strategy for Patient-Oriented Research (SPOR) Evidence Alliance (SPOR EA 2021) to identify potential trainees and match them with mentors to enhance capacity in knowledge synthesis methods in drug safety and effectiveness research. MAGIC focuses on building capacity across three educational streams—Stream 1: Graduate-level trainees and fellows from various disciplines (e.g., epidemiology, biostatistics, clinical epidemiology, health economics) interested in health technology assessment and knowledge synthesis science; Stream 2: researchers and research staff who are interested in gaining familiarity with the principles of knowledge synthesis and NMA; and Stream 3: knowledge users including policy makers, clinicians, and patients interested in enhancing their skills to use, appraise, and interpret knowledge synthesis and NMA. Courses and learning opportunities are tailored to each stream (Box 1).

MAGIC has demonstrated substantial productivity in addressing queries. To date, we responded to 62 queries (54% of all DSEN queries) that are completed/ongoing (Supplementary Table S1). Since 2014, MAGIC produced 26 reports for knowledge users and 49 publications (DSEN MAGIC 2021). MAGIC team members completed an additional 92 knowledge synthesis methods papers that were not DSEN funded, but that advanced and informed methods. A number of papers describing methodologic advances were also completed by the MAGIC team including reporting guidelines (e.g., for scoping reviews and NMAs (Hutton et al. 2015; Tricco et al. 2016a; Tricco et al. 2018)) and presentation of NMA (Veroniki et al. 2016) results. An additional >$99.7 million in peer-reviewed research funding was held by MAGIC members. A total of 125 national and 88 international presentations were led by MAGIC team members. Team members mentored >100 trainees toward developing capacity in the next generation of DSEN researchers. Four trainees participated in DSECT. Nine trainees received prestigious awards (e.g., Polanyi Prize, Vanier Scholar, CIHR Banting Doctoral Award, CIHR Rising Star Awards), and three of these now have university affiliations. More than 10,000 learners have engaged in the various courses MAGIC investigators have provided including >100 patients in our Partners in Research course. A variety of research support tools have been created including “What review is right for you”, which matches the query to a review method and the rank heat plot (Veroniki et al. 2016), which plots the results from all NMA outcomes into one figure.

Creating a pipeline of researchers in Canada is critical to ensuring continued development and innovation in drug safety and effectiveness and ensuring the foundations are in place for Canada to maintain national and international reputation in this space. In addition to Drs. Tricco and Hutton’s success with the very competitive CIHR New Investigator Awards, Dr. Tricco now holds a Tier 2 Canada Research Chair in Knowledge Synthesis. Drs. Straus and Tricco supervised post-doctoral fellow Dr. Veroniki who received a Banting Post-doctoral Award and she was awarded the Polanyi Prize for her methodologic contributions. Drs. Straus, Tricco, and Veroniki then supervised PhD student, Dr. Watt. Dr. Watt received the CIHR Rising Star Award for her research and is currently a scientist in the KT Program at St. Michael’s Hospital and a faculty member of the University of Toronto. MAGIC trainees have been employed in academia, industry, and government, highlighting the breadth of expertise these trainees have.

Ultimately, DSEN was developed to inform practice and policy and MAGIC has met this goal. Specifically, our goal is to improve the health of all Canadians by influencing at least one policy decision per year as measured by a tangible change in policy or action by a DSEN knowledge user that is directly attributable to our research; this outcome was determined based on feedback obtained from the knowledge users that posed the query. Our goal of influencing at least 1 policy decision per year was surpassed. For example, the British Columbia Ministry of Health commissioned a systematic review and NMA on long-acting and intermediate-acting insulin. Findings were used to continue listing coverage of these agents in a similar manner and the report was published in the BMJ (Tricco et al. 2014). MAGIC was later contacted by the World Health Organisation (WHO), which requested a report based on this work for their Essential Medicines initiative (a program that provides recommendations to countries worldwide regarding which medications should be provided to their citizens). The WHO subsequently requested an expansion of the query to include biosimilars, which was also of interest to Health Canada and CADTH. As such, a second essential medicines application was submitted and identified by the WHO as being the second most important application in the world (Tricco et al. 2021a, 2021b). This query shows MAGIC’s ability to use an iterative approach, while fostering both national and international collaborations to achieve global impact.

As another example, CADTH commissioned MAGIC to explore the efficacy and safety of interferon-free direct-acting antiviral agents for chronic hepatitis C infection. This review was commissioned on behalf of provincial and territorial public drug plans. This query presented a unique methodological challenge as novel study designs were permitted by regulators for the newer treatment regimens. These study designs included interventional, single-arm studies (i.e., no formal comparative control group included in the design or a historical control cohort was used) or studies where only a single arm of the study fits the eligibility criteria. The NMA methodology was adjusted to incorporate the effect estimates from this single-arm evidence into the networks of treatments. It was not possible to use comparative effectiveness methods (e.g., propensity score weighting) as no individual patient data were available. Instead, we incorporated single-arm studies into the NMA by creating a “virtual” study where a comparator arm matched for baseline patient characteristics was identified for the single arm. The results of the review were presented to the Canadian Drug Expert Committee (CDEC) and, together with the cost-effectiveness analysis, CDEC made recommendations, which were disseminated to the provincial and territorial decision-makers. Both a publicly available technical report and a peer-reviewed article were generated (CADTH 2016; Wong et al. 2017).

The Ministry of Health in Nunavut requested a rapid review of interventions for latent tuberculosis prophylaxis. This review was later updated and results were used to inform their policy crisis (Pease et al. 2017; 2018).

These syntheses show the diverse range of review methods available within MAGIC. Moreover, MAGIC uses modified methods to meet urgent timelines (such as during the COVID-19 pandemic) to ensure timely decision-making. Indeed, MAGIC has lead research on optimising rapid review methods, including use of automation and machine learning (Pham et al. 2021).