Evaluating prospective study registration and result reporting of trials conducted in Canada from 2009 to 2019

Publication bias is “the tendency on the part of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings” (Dickersin 1990). Studies with statistically significant or positive results are more likely to be published than those with statistically non-significant or negative results—this means the dissemination of research findings is a biased process (Stern and Simes 1997; Dubben and Beck-Bornholdt 2005; Song et al. 2010; Dwan et al. 2013). Similarly, studies with positive results were much more likely to be published in a shorter time than studies with indefinite conclusions (Stern and Simes 1997). Publication bias threatens the practice of evidence-based medicine, the validity of meta-analyses, and the reproducibility of a study (Marks‐Anglin and Chen 2020). Under-reporting due to publication bias exaggerates the benefits of treatments and underestimates their harms (McGauran et al. 2010). Ultimately, this is detrimental to the healthcare system, as it wastes resources and puts patients at risk (Moher 1993; Chalmers et al. 2013). Between 1999 and 2007, fewer than half of all trials registered and completed on ClinicalTrials.gov were published (Ross et al. 2009). Between 2010 and 2012, fewer than half of all the registered trials for rare diseases were published within four years of their completion (Rees et al. 2019).

In 2008, the World Medical Association's (WMA) Declaration of Helsinki stated that “every clinical trial must be registered in a publicly accessible database before recruitment of the first subject” (WMA 2021). The declaration also states that all studies should be published, regardless of the statistical significance of their outcome. Nonetheless, publication bias remains prevalent globally. For instance, in all 36 German university medical centers between 2009 and 2013, only 39% of clinical trials were published within 2 years of their completion (Wieschowski et al. 2019). In 2015, the World Health Organization (WHO) published a Statement on Public Disclosure of Clinical Trial Results. It states that the main findings of clinical trials are to be published at the latest 24 months after study completion and that “the key outcomes are to be made publicly available within 12 months of study completion by posting to the results section of the primary clinical trial registry” (WHO 2022). This WHO statement serves as a global guideline to reduce publication bias and improve overall evidence-based medical decision-making.

In 2020, the Canadian Institutes of Health Research (CIHR) asserted that, as of 2021, it would implement new policies to ensure full adherence to the WHO Joint Statement requirements (Government of Canada CI of HR 2020). Similarly, the Canadian Government recommends adherence to the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2), which states that all clinical trials shall be registered before recruitment of the first trial participant and that researchers shall promptly update the study registry with the location of the findings (TCPS 2018). The Canadian Government also encourages sponsors to register clinical trials in a publicly accessible registry such as ClinicalTrials.gov (Canada 2003). Despite these recommendations, explicit guidance for trial reporting is lacking (Cobey et al. 2017). Even in jurisdictions where the legal frameworks have been established around trials registration and reporting (e.g., Food and Drug Administration in the United States), low adherence persists (DeVito et al. 2018).

This research aimed to evaluate the adherence of clinical trials conducted in Canada to study registration and reporting best practices. Specifically, among interventional trials registered on clinicaltrials.gov and completed between 2009 and 2019, we aimed to evaluate the proportion of trials that were prospectively registered on clinicaltrials.gov (Dickersin 1990); the proportion of trials with study results reported on clinicaltrials.gov (Song et al. 2010); the proportion of trials with publication of findings (Stern and Simes 1997); and characteristics related to clinical trials that may predict prospective registration, reporting of study results, and publication of findings (Dwan et al. 2013).

We calculated the proportions of trials that were prospectively registered, had their results reported in the registry, and subsequently had their findings published.

We analyzed whether the results were modified by these 10 variables of interest as outlined in the study characteristics above using univariable logistic regressions (odds ratios with 95% confidence intervals). This analysis enables us to identify which characteristics have the strongest impact on best practice adherence. The outcomes of this analysis were reported as a prevalence in percentages.

A total of 6790 clinical trials conducted in Canada were identified. Of those clinical trials, we excluded 70 of them that were submitted to clinicaltrials.gov with one or more incorrect entries and were therefore incompatible with our software (Microsoft Excel). For example, among those 70 studies, a few submitted an “alphabetic” entry despite a strictly “numerical” requirement. Our software could not correct those mistakes, and therefore those studies were excluded. Therefore, a total of 6720 (99%) studies were included in our analysis.

Of the almost 7000 RCTs in our sample, less than two-thirds were prospectively registered; less than half made their results available; and less than two-thirds published their findings. Less than a quarter of the RCTs completed all three best practices. Trials conducted with exclusively Canadian sites were substantially less compliant with these practices than trials with both Canadian and international sites. Importantly, the trials we describe were not small (Chan and Altman 2005); nearly a third of the trials included between 100 and 500 participants, and 15% (n = 1041) of them included over 500 participants.

Of all the variables included in the logistic regression, four were highly associated with study registration and reporting best practices. They were as follows: clinical trials led by “Industry” (pharmaceutical companies) (Dickersin 1990), phase 3 clinical trials (Song et al. 2010), trials with over 500 participants (Stern and Simes 1997), and, finally, clinical trials conducted by a multinational team (Dwan et al. 2013). The four variables that were negatively associated with study registration and reporting best practices were as follows: clinical trials led by “Academia” (universities) (Dickersin 1990), phase 1 clinical trials (Song et al. 2010), trials with fewer than 100 participants (Stern and Simes 1997), and trials conducted with exclusively Canadian sites (Dwan et al. 2013).

Some readers will view these results as another example of egregious waste in biomedicine with little improvement since the 2014 landmark Lancet series on research waste (Kleinert and Horton 2014). Patients, who are critical to the success of clinical trials, are likely to be disappointed with these results; their contributions are not being honored. For clinical practice guideline developers, these results indicate that evidence is missing regarding the totality of knowledge about an intervention. For healthcare funders, these results indicate a bad return on investment. If grantees use scarce resources, often taxpayer dollars, and do not prospectively register their trials and/or make their results available in registries or publish their results, everyone loses. Finally, academic institutions may risk their institutional reputation when their faculty members fail to meet minimum national/global standards (WHO; CIHR).

These results are not unique to Canada. Similar results have been reported elsewhere (van den Bogert et al. 2016; Rüegger et al. 2017; Wieschowski et al. 2019; Taylor and Gorman 2022). The overall lack of prospective registration and reporting of clinical trial results may reflect a lack of knowledge on the topic on the part of clinical trials teams and/or their academic institutions. Prospective registration and reporting results are part of a larger ecosystem of open science, which includes transparency. Compared with some other parts of the world, Canada has been slow to publicly embrace the practices of open science (Government of Canada 2022).

The lack of prospective registration may reflect that clinical trial principal investigators are leaving these responsibilities to other team members. Alternatively, funders may not have strong enough adherence monitoring in place. With the advent of automated digital monitoring and the increasing availability of application programming interfaces (APIs) this should be less of a problem. The European Trials Tracker scheme, developed by the University of Oxford's Bennett Institute for Applied Data Science, is an example of existing monitoring on a large scale (https://www.bennett.ox.ac.uk/) (Bennett Institute for Applied Data Science 2022). Funders and academic institutions could use digital dashboards to monitor adherence to clinical trial policies and identify training needs (Cobey et al. 2022).

In the last few years, several initiatives have proposed moving away from traditional metrics of the number of publications (the irony of our results—counting publications—is not lost on us) towards a broader set of best practices that reflect an institutional commitment to research integrity when assessing researchers for hiring, promotion, and tenure (Hicks et al. 2015; Moher et al. 2020; DORA 2022). Current researcher assessment could be augmented by tracking whether faculty members have prospectively registered their trials (and other studies), made their results available on a trial register, and published them (preferably in an open access journal).

It is time for trial funders and academic institutions to collaborate to address the overall lack of adherence of Canadian trials to registration and reporting mandates. In the Canadian context, there is now a requirement for equity, diversity, and inclusion training when applying for grant funding for the CIHR. CIHR has had success in requiring principal investigators to take sex and gender training before they can submit a grant application (Haverfield and Tannenbaum 2021). Something similar could be introduced for clinical trial registration and results reporting. The training can be required for all faculty and research staff involved in conducting trials. This would portray the funders and academic institutions’ commitment to improving this situation. They could also collectively commit to evaluating such an educational intervention by conducting a stepped wedge and/or cluster trial across universities to ensure the training was having the desired effect.

From the earliest attempts to introduce clinical trial registration in the 1980s, there was a strong belief that it might reduce publication bias and provide a more accurate picture of the estimated benefits and harms of interventions. Our results, and others (Scherer et al. 2018; Wieschowski et al. 2019), suggest that publication bias is still a substantive problem despite prevalent views that clinical trials are heavily regulated. Indeed, although regulation via policy exists, if we fail to audit adherence, the policy goals will not be adhered to.

CIHR recently updated their policy guide (see box). The “stick” in this updated guidance is likely to be the lack of future funding for principal investigators unless their current trials are registered, and the results made available. Importantly, the agency will monitor adherence annually “by asking impacted researchers to provide clinical trial registration identifiers, and links to summary results and open access publications”.

Nominated principal investigators receiving CIHR grant funds for clinical trial research after 1 January 2022 must comply with the above requirements to remain eligible for any new CIHR funding”. Despite the recently updated policy guide, the CIHR stated that in 2022, only 57% of the CIHR-funded trials “had registered their clinical trial in a publicly available, free to access, searchable clinical trial registry complying with WHO's international agreed standards before the first visit of the first participant” (Government of Canada CI of HR 2023). Perhaps, adherence to the new policy guide may only be realized for future CIHR-funded trials once nominated principal investigators lose funding eligibility for their lack of compliance.

We relied on the data reported on clinicaltrials.gov. This is the most widely used registration platform (Zarin et al. 2011), accounting for the vast majority of all clinical trial registrations. To be counted in our analysis, we only tracked forward the trials that were marked as “completed” on clinicaltrials.gov; this means we missed records that were registered and then never updated to be marked as complete despite these trials having ended. The results are likely worse than we report here. Moreover, we only tracked studies that were registered on clinicaltrials.gov in the first place. Had we also analyzed publications that were not registered on clinicaltrials.gov, adherence may have been lower. Furthermore, we did not analyze the length of time it took between the study completion date and the publication date. In other words, some studies may have posted their results and published their findings a decade after the study completion, despite the recommendation to do so within 2 years (WHO 2022). Despite all of this, less than one-quarter of the sample adhered to all three best practices. Some may argue that trials with non-statistically significant results take longer to publish; however, a recent review found no difference in time to publication based on the statistical significance of the trial (Jefferson et al. 2016).

Another limitation is that we relied on clinicaltrials.gov regarding the publication of study findings. Ideally, we would have completed the quality assurance for our entire sample; however, due to a resource constraint, we limited the quality assurance to 10% of our sample. As demonstrated by our quality assurance, clinicaltrials.gov has underestimated the true prevalence of publications by roughly 14%. Clinicaltrials.gov reported that 56% (n = 378) of trials were published, but our quality assurance determined the true number to be 70% (n = 470). This is in line with a recent study highlighting that over a third of trials they analyzed on clinicaltrials.gov had no results available on either clinicaltrials.gov or PubMed, up to 36 months after their primary completion date (Nelson et al. 2023). Some publications were missed by clinicaltrials.gov for the following reasons: publication does not include NCT IDs; publication date was after we collected the data; publications were in the form of an Abstract/Thesis/Poster; and, finally, some publications were in journals not accessible by clinicaltrials.gov. Overall, this underestimation of publication status suggests that despite low levels of prospective registration and results reporting, more results from these trials are being published. All three best practices must be adhered to concurrently to maintain a high level of scientific rigor. If findings are published without prospective registration or results reporting, a bias is introduced into the published paper. Therefore, this underestimation of publication status supports our findings that there is room for improvement. Ultimately, our analysis is limited by how comprehensive and detailed the documentation is on clinicaltrials.gov.

In summary, our analysis of nearly 7000 Canadian trials registered on clinicaltrials.gov indicates that there is substantial room for improvement in ensuring they are prospectively registered at inception (prior to the first person being randomized), the results are reported in a publicly accessible registry, and that completed trials are published, preferably in an open access platform/journal. The consequences of not monitoring adherence to these activities are profound and wasteful. The international AllTrials initiative, signed by more than 700 organizations, has brought attention to public support for these policies (AllTrials 2022). Several Canadian-based foundations have signed this declaration (e.g., Canadian Cancer Research Alliance, Canadian Agency for Drugs and Technologies in Health, Canadian Medical Association, and Canadian HIV Trials Network). Other stakeholders, like Health Canada, and funders of academic trials, like the federal Tri Agency, ought to also commit to the AllTrials initiative and its broader principles. Canada should join the global movement to address research waste due to incompliant registration and reporting of trials and seek to lead in identifying resolutions.