Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian Institutes of Health Research Project Grant Competition
Abstract
The Canadian Institutes of Health Research (CIHR) commenced a Quality Assurance Program in 2019 to monitor the quality of peer review in its Project Grant Competition Peer Review Committees. Our primary aim was to describe the performance of CIHR grant peer reviewers, based on the assessments made by CIHR peer review leaders during the first 3 years of the Research Quality Assurance Program. All Peer Review Committee Chairs and (or) Scientific Officers who led peer review for CIHR in 2019, 2020, and 2021 completed Reviewer Quality Feedback forms immediately following Peer Review Committee meetings. The form assessed Performance, Future potential, Review quality, Participation, and Responsiveness. We summarised and descriptively synthesised data from assessments conducted after each of the four grant competitions. The performance of peer reviewers on 4438 occasions was rated by Chairs and Scientific Officers. Approximately one in three peer reviewers submitted outstanding reviews or discussed additional applications and one in 10 demonstrated potential as a future Peer Review Committee leader. At most, one in 20 peer reviewers was considered to have not performed adequately with respect to review quality, participation, or responsiveness. There is a need for more research on the processes involved in allocating research grant funding.
Introduction
Research funding is a major determinant of what scientists study, yet there has been relatively little research into the various methods that major funders use to award competitive grants. Evidence should inform how funders design, deliver, monitor, and evaluate grant peer review (Shepherd et al. 2018), yet very few national health research funders have evaluated the quality of their grant peer review.
The European Science Foundation Survey Analysis Report on Peer Review Practices mentions 25 funders that evaluate the quality and useability of grant peer review reports as standard practice (European Science Foundation 2011a). A small number of funders appear to use formal quality assurance processes covering quality indicators or expectations of quality peer review/peer reviewers. Some funders may commission external auditors to conduct internal quality assurance of their processes. Quality assurance is nominated as a central pillar of best practice in grant peer review (European Science Foundation 2011b). Several tools have been developed to assess the quality of journal peer review, although none have sufficient validity to warrant widespread use at this time (Superchi et al. 2019). We are unaware of published tools for assessing the quality of grant peer review.
Systematic evaluation keeps funders accountable to the communities they serve and demonstrates commitment to quality improvement (European Science Foundation 2012). In 2019, after a 2-year pilot phase, the Canadian Institutes of Health Research (CIHR) commenced a continuous quality assurance program (Research Quality Assurance Process) to routinely monitor the quality of peer reviewers’ participation in and contributions to Project Grant Competition Peer Review Committees (Canadian Institutes of Health Research 2022e). Peer review leaders (committee Chairs and Scientific Officers) evaluated the performance of each Peer Review Committee member contributing to delivering CIHR peer review, according to criteria developed and refined by CIHR over the pilot phase.
Regular audit of performance would help to establish whether funders meet their aspirations in delivering peer review (European Science Foundation 2011b). Yet, few funders institute formal monitoring of the quality of grant peer review (e.g., quality assurance) (European Science Foundation 2011a; Guthrie et al. 2018; Recio-Saucedo et al. 2022).
The primary aim of this study was to describe the performance of grant peer reviewers in the Canadian health research funding system, based on the assessments made by CIHR peer review leaders, during the first 3 years of the Research Quality Assurance Process. The secondary aims were to describe the performance of grant peer reviewers by sex, career stage, university affiliation, and geographic location.
Context
Grant peer review comes in various forms including (i) an expert committee reviewing all grants under consideration and rating or ranking them, (ii) each grant being sent to a small number of expert reviewers (1–2 people) whose feedback may provide a final score, or contribute to an expert panel's grant discussions and ranking, and (iii) a randomised (lottery) component once certain criteria are met (Liu et al. 2020; Horbach et al. 2022). The peer review committees that were assessed for CIHR's Research Quality Assurance Process operated as expert committees that reviewed all grants and rated or ranked them.
The CIHR Project Grant program—which accounts for approximately $750 million of CIHR's $1.3 billion annual funding budget—is open to researchers at any career stage who wish to conduct health-related research. Peer Review Committees cover the breadth of research under CIHR's mandate of (i) biomedical, (ii) clinical, (iii) health systems and services, and (iv) population health research. Prior to 2020, committee meetings were held exclusively in a face-to-face setting; since then, all committee meetings have been conducted virtually using Microsoft Teams.
Review Committee preparation and training
Prior to serving on a Peer Review Committee, reviewers are expected to study the CIHR funding policies and guidelines, agree to abide by the CIHR Confidentiality and Conflict of Interest Policy (Canadian Institutes of Health Research 2022b) and Standards of Practice for Peer Review (Canadian Institutes of Health Research 2022a), read and understand the CIHR Guide on Handling Documents used in Peer Review (Canadian Institutes of Health Research 2022d), and complete online training (including learning modules on unconscious bias and sex- and gender-based analysis).
Peer review process
Each CIHR Peer Review Committee comprises a Chair and a Scientific Officer (committee leaders) and up to 20 committee members (peer reviewers). The Chair and Scientific Officer, with support from CIHR staff, assign each application to three reviewers who review the application in detail—each reviewer receives up to 10 applications per competition. Each committee handles a median 40 applications per competition. The Peer Review Committees are responsible for (i) evaluating and rating each application they are assigned by discussing the strengths and weaknesses of the applications and (ii) considering the budget and project term for applications that are discussed. The CIHR Standards of Practice for Peer Review outlines the roles, responsibilities, and expectations of Peer Review Committee members (Canadian Institutes of Health Research 2022a).
Methods
This is a descriptive retrospective analysis of quality assurance data that were collected at the conclusion of four CIHR Project Competition Peer Review Committee meetings (fall 2019, fall 2020, spring 2021, fall 2021). We analysed data that were collected as part of routine, continuous quality improvement efforts initiated by CIHR to measure, monitor, encourage, and support grant peer reviewer participation and performance. The Behavioural Research Ethics Board at the University of British Columbia approved the study (project ID H21-03906; approved 28 January 2022).
CIHR's Quality Assurance Program
CIHR developed a standard approach to monitoring and evaluating the quality of its peer review. The program includes developing policies, guidelines, and tools for measuring and monitoring peer review, and evaluating and continuously improving how peer review is delivered across the agency. The Research Quality Assurance Feedback form (Table 1) was developed by CIHR in 2017 to assess Peer Review Committee member performance; its format, content, and construction were underpinned by evidence from four sources: (i) literature scan, (ii) benchmarking analysis, (iii) internal document review, and (iv) feedback from stakeholders.
Table 1.
| Effective contributor | Outstanding contributor | Needs improvement | Other considerations | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Y/N | Review quality | Review quality | Participation | Review quality | Participation | ||||||
| Y/N | Participation | Y/N | Reviews went over and above expectations (e.g., insightful comments—detailing both strengths and weaknesses—that will significantly aid both the applicant and other reviewers) | Y/N | Constructively participated in application discussions NOT assigned to them | Y/N | Review(s) lack appropriateness (e.g., contain biased and (or) inappropriate comments) | Y/N | Lacks professionalism (e.g., stubborn, combative, overly critical/vocal, and inappropriate/biased comments) | Y/N | Demonstrates potential as future Chair |
| Y/N | Agreed to take on additional tasks on short notice (e.g., to review more applications) | Y/N | Review(s) lacks robustness (e.g., contain insufficient details to justify given ratings) | Y/N | Difficult to chair (e.g., not responsive to Chair's suggestions, interrupts, and speaks too much) | Y/N | Demonstrates potential as future Scientific Officer | ||||
| Y/N | Review(s) lack utility (e.g., comments are not constructive and not helpful to applicants to improve their future submissions) | Y/N | Major presentation weaknesses (e.g., not well prepared, difficulty presenting, difficult to hear, and unclear/unfocused comments) | Y/N | Demonstrates potential as future Peer Reviewer Mentor | ||||||
| Y/N | Low participation level (e.g., not engaged as primary reviewer in discussions and absentminded) | ||||||||||
Note: Peer reviewers who recuse themselves from a committee discussion for conflicts of interest are not assessed.
Developing the Research Quality Assurance Feedback form
In fall 2015, CIHR staff scanned PubMed and Google Scholar to identify publications addressing peer review quality. The search identified 16 articles (Table S1), the results of which were synthesised into three general criteria for assessing peer review quality: appropriateness, robustness, and utility. At the same time, CIHR undertook a benchmarking analysis of national and international funders, scanning the websites of 49 funders. Nine funders described a review quality assurance process with quality indicators, standards, or expectations (Table S2).
Next, CIHR reviewed its policy documents intended to guide peer reviewers on expectations, conflicts of interest, and confidentiality policies. Finally, CIHR conducted consultation sessions with stakeholders to review the draft review quality criteria, quality indicators, and Feedback form. Stakeholders recommended two additional indicators for the robustness criterion (credible comments and appropriate reviewer assignment). Feedback from a working group of experienced Peer Review Committee leaders guided the current format of the feedback form and the instructions for completing the form.
Staff routinely analyse data collected through the Quality Assurance Program and have a formal procedure in place for systematic follow-up with peer reviewers if review quality or participation issues are flagged by the Peer Review Committee leaders. Staff may provide additional training and (or) arrange mentoring for peer reviewers identified as having problems with review quality, participation, or responsiveness.
Assessing the quality of the peer review
The quality of each reviewer's contribution was assessed across three domains using the Reviewer Quality Feedback form: (1) the quality of the written and oral presentation of reviews in relation to appropriateness, utility, and robustness, (2) participation in the discussion at the peer review meeting as professional, responsive, and engaged, and 3) potential to fulfill the role of Chair or Scientific Officer in future meetings. All checklist items were assessed as Yes or No. Additional guidance was available to Chairs and Scientific Officers to help them interpret the Reviewer Quality Feedback form items related to appropriateness, robustness, and utility (Table 2).
Table 2.
| Review quality criterion | Definition | Interpretation |
|---|---|---|
| Appropriateness | Review comments are fair, understandable, confidential, and respectful | • Review respects CIHR's Conflict of Interest and Confidentiality policy • Absence of comments that suggest bias against the applicant(s) due to sex, ethnicity, age, language, career stage, institutional affiliation, or geographic location • Review is original, and written in clear and understandable language • Absence of comments that can be construed as sarcastic, flippant, or arrogant |
| Robustness | Review is thorough, complete, and credible | • Review contains a detailed justification of each rating, including meaningful and clearly expressed descriptions of both the application strengths and weaknesses • Comments align with the given rating • Review addresses all applicable adjudication criteria and does not include information that is not relevant to the adjudication criteria • All comments on grant content are factually correct • Absence of statements that could call into question the reviewer's scientific knowledge or expertise |
| Utility | Review provides feedback that addresses the needs of reviewers, applicants, and funders | • Review comments are constructive and may help applicants to improve their future submissions and (or) advance their research • Review contains information that allows other reviewers to understand the reviewer's rating(s) • Review is detailed enough to be used by CIHR to evaluate and refine review process elements |
Participants
All Peer Review Committee leaders who contributed to a Peer Review Committee in 2019, 2020, and 2021 for the CIHR Project Grant Competition rounds assessed the performance of Peer Review Committee members (peer reviewers). Independent researchers at all career stages are eligible to serve on Peer Review Committees; they self-declare their career stage based on the year that they first met CIHR's definition of an independent researcher (autonomous regarding their researcher activities, and in an academic or research appointment (Canadian Institutes of Health Research 2022c)).
Peer Review Committee members were recruited from one of the following:
i.
An established database of researchers and experts from Canadian and international institutions who have previously participated in CIHR peer review (College of Reviewers).
ii.
Nominations from existing Peer Review Committee Chairs and Scientific Officers.
iii.
Internet search by CIHR program staff (including Canada Research Chairholders list, Fellows Directory for the Canadian Academy of Health Sciences, publications, conference-invited speakers, and targeted searches of institutions in regions that are under-represented on committees).
Peer review committee member characteristics
Demographic information (sex, self-declared career stage, university affiliation, and geographic location) and history of contributions to previous CIHR grant competition rounds (i.e., the number of times the person had submitted a CIHR grant application, and the number of times the person had served as a Peer Review Committee member) are collected and retained by CIHR for all registered applicants and peer reviewers to facilitate efficient communication with the individuals who engage with CIHR.
Process for obtaining consent
All Chairs, Scientific Officers, and peer reviewers who participate in the CIHR Project Grant Competition consent to the CIHR Standards of Practice for Peer Review (Canadian Institutes of Health Research 2022a) prior to completing their grant peer review assignments. By consenting to the Standards of Practice for Peer Review, participants agree to (1) assessment of their performance as a grant peer reviewer by the Chair and (or) Scientific Officer of the Peer Review Committee, and to (2) receiving feedback, mentoring, and (or) resources regarding review quality, participation, and responsiveness from CIHR staff.
All who contribute to Peer Review Committees are informed that the data collected about their performance are routinely collected and analysed as part of CIHR's core business of delivering health research grant peer review across Canada, and that this may include publishing aggregate data.
Procedures
Immediately following Peer Review Committee meetings, Chairs and Scientific Officers completed the Reviewer Quality Feedback form online (Microsoft Excel) for each committee member. CIHR staff collected and collated the completed checklists, and checked for missing data. A staff member, who was present in the peer review meeting, reviewed each completed checklist to validate the responses to the participation items. The staff contacted Chairs and Scientific Officers to follow-up any missing data, as required. If Chairs and Scientific Officers assessed the performance of the Peer Review Committee (i.e., not the individual peer reviewers), these data were excluded from the final analysis of quality, participation, and other items.
Data analysis and synthesis
We summarised and descriptively synthesised data from assessments conducted after each of the four grant competitions, separately. We calculated and presented in star plots, the proportion of committee members (i) whose reviews exceeded expectations, (ii) who participated constructively in discussions of applications to which they were not assigned, (iii) who completed additional tasks at short notice, (iv) who demonstrated potential as a future Chair, Scientific Officer, or Peer Reviewer Mentor, (v) whose review(s) lacked robustness, (vi) whose reviews lacked appropriateness, (vii) with low participation, (viii) with major presentation weakness, (ix) who were difficult to chair, and (x) who lacked professionalism. We used Microsoft Excel to build the database and produce the summary results and Flourish (https://flourish.studio/) for data visualisation.
Next, we stratified the data by subgroups to explore peer reviewer performance by sex, career stage (early career versus not), university affiliation (member of the U15 Group of Canadian Research Universities (the U15 describes itself as “a collective of some of Canada's most research-intensive universities” (U15 Group of Canadian Research Universities 2022); the group claims to receive 80% of all competitive research funding in Canada (U15 Group of Canadian Research Universities 2022)) versus others), and geographic location (Canada versus international).
Results
The performance of peer reviewers was evaluated on 4438 occasions (59 committees with 991 peer reviewers in fall 2019, 57 committees with 1123 peer reviewers in fall 2020, 61 committees with 1230 peer reviewers in spring 2021, and 57 committees with 1094 peer reviewers in fall 2021). There were 2459 unique reviewers who participated across the four competition rounds, slightly more than half of the reviewers (n = 1269, 52% of 2459) participated in two or more peer review rounds. 1190 (48% of 2459) reviewers participated in one Project Grant Competition round, 698 (28% of 2459) reviewers participated in two rounds, 434 (18% of 2459) reviewers participated in three rounds, and 139 (6% of 2459) reviewers participated in all four peer review rounds.
Review Quality Feedback forms were returned for all Peer Review Committees in every competition round (Table 3). In the 2021 competitions, between four and six Chairs and Scientific Officers completed one feedback form for the Peer Review Committee (i.e., did not complete the Peer Review Feedback form for individual peer reviewers); these data are not included in the calculations of proportions of committee members for quality, participation, or future potential criteria. The number of Chairs and Scientific Officers who did not provide any feedback for committee members is also presented in Table 3. Demographic data for Chairs and Scientific Officers in each Project Grant Competition are presented in Table 4.
Table 3.
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
|---|---|---|---|---|
| Chairs and Scientific Officers (n) | ||||
| Participants | 167 | 174 | 181 | 172 |
| Provided feedback for individual reviewers | 108 | 152 | 158 | 154 |
| Provided feedback for committee only | 0 | 0 | 4 | 6 |
| No feedback provided | 59 | 22 | 19 | 12 |
| Peer Review Committees (n) | ||||
| Total committees | 59 | 57 | 61 | 57 |
| Committees with complete RQA data | 59 | 57 | 61 | 57 |
Table 4.
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
|---|---|---|---|---|
| n = 108 | n = 152 | n = 158 | n = 154 | |
| Sex (n, %) | ||||
| Female | 48 (44) | 78 (51) | 84 (53) | 86 (56) |
| Male | 60 (56) | 74 (49) | 74 (47) | 68 (44) |
| Career stage (n, %) | ||||
| Early | 3 (3) | 1 (<1) | 1 (<1) | 5 (3) |
| Mid | 36 (33) | 61 (40) | 47 (30) | 58 (38) |
| Senior | 66 (61) | 84 (55) | 102 (65) | 87 (56) |
| Missing | 3 (3) | 6 (4) | 8 (5) | 4 (3) |
| Affiliation (n, %) | ||||
| U15 | 81 (75) | 113 (74) | 113 (72) | 114 (74) |
| Non-U15 | 17 (16) | 23 (15) | 24 (15) | 22 (14) |
| Missing | 10 (9) | 16 (11) | 21 (13) | 18 (12) |
| CIHR College of Reviewers member (n, %) | ||||
| Yes | 101 (94) | 125 (82) | 132 (84) | 133 (86) |
| No | 7 (6) | 27 (18) | 26 (16) | 21 (14) |
Note: U15, institution is a member of the U15 Group of Canadian Research Universities.
Peer review committee composition
Of the 4438 peer review contributions evaluated, there were slightly more by early-career women than early-career men, more by mid-career men than mid-career women, and more by senior-career men than senior-career women (Fig. 1) over the study period. Peer reviewers each completed a median seven review assignments for each Project Grant Competition, had participated in peer review for a median of one Project Grant Competition in the 12 months prior to the competition for which their performance was assessed, and had received a median of two grants as principal investigator in the 3 years prior to the Project Grant Competition for which their performance was assessed (Table 5).
Fig. 1.
Table 5.
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
|---|---|---|---|---|
| n = 991 | n = 1123 | n = 1230 | n = 1094 | |
| Sex (n, %) | ||||
| Female | 408 (41) | 485 (43) | 511 (42) | 424 (39) |
| Male | 583 (59) | 638 (57) | 715 (58) | 665 (61) |
| Missing | 0 | 0 | 4 (<1) | 5 (<1) |
| Career stage (n, %) | ||||
| Early | 59 (6) | 99 (9) | 132 (11) | 140 (13) |
| Mid | 384 (39) | 456 (41) | 494 (41) | 421 (38) |
| Senior | 502 (51) | 523 (47) | 535 (43) | 458 (42) |
| Missing | 46 (5) | 45 (4) | 69 (6) | 75 (7) |
| Affiliation (n, %) | ||||
| U15 | 834 (84) | 960 (85) | 1044 (85) | 918 (84) |
| Non-U15 | 157 (16) | 163 (15) | 186 (15) | 176 (16) |
| CIHR College of Reviewers member (n, %) | ||||
| Yes | 837 (84) | 969 (86) | 1008 (82) | 872 (80) |
| No | 102 (10) | 141 (13) | 205 (17) | 222 (20) |
| Missing | 52 (5) | 13 (1) | 17 (1) | 0 |
| Location | ||||
| Canada | 968 (98) | 1098 (98) | 1200 (98) | 1076 (98) |
| International (n, %) | 23 (2) | 25 (2) | 30 (2) | 18 (2) |
| Project grant applications assigned^ (n) | ||||
| Min | 1 | 1 | 1 | 1 |
| Max | 11 | 10 | 10 | 10 |
| Median | 7 | 7 | 6 | 6 |
| Competitions serving as peer reviewer* (n) | ||||
| Min | 0 | 0 | 0 | 0 |
| Max | 3 | 4 | 3 | 3 |
| Median | 1 | 1 | 1 | 1 |
| Grant funding history as Principal Investigator∼ (n) | ||||
| Min | 0 | 0 | 0 | 0 |
| Max | 7 | 8 | 8 | 8 |
| Median | 2 | 2 | 2 | 2 |
Note: U15, institution is a member of the U15 Group of Canadian Research Universities.
^
In the competition.
*
In the 12 months prior to the competition.
∼
Number of grants held in the 3 years prior to the competition.
Approximately one in three peer reviewers submitted outstanding reviews or discussed additional applications and one in 10 peer reviewers demonstrated potential as a future Chair, Scientific Officer, or Peer Reviewer Mentor (Fig. 2;Table 6).
Fig. 2.
Table 6.
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
|---|---|---|---|---|
| n = 991a | n = 1123b | n = 1230c | n = 1094d | |
| Performance | ||||
| Undertook additional tasks, n (%) | 51 (5.1) | 23 (2.0) | 47 (3.8) | 39 (3.6) |
| Discussed additional applications, n (%) | 309 (31.2) | 430 (38.3) | 380 (30.9) | 311 (28.4) |
| Outstanding review, n (%) | 285 (28.8) | 487 (43.4) | 386 (31.4) | 356 (32.5) |
| Future potential | ||||
| Committee Chair, n (%) | 136 (13.7) | 172 (15.3) | 141 (11.5) | 112 (10.2) |
| Committee Scientific Officer, n (%) | 171 (17.3) | 196 (17.5) | 171 (13.9) | 171 (15.6) |
| Potential Peer Reviewer Mentor, n (%) | 145 (14.6) | 209 (18.6) | 236 (19.2) | 160 (14.6) |
Note: N = number of peer reviewers who contributed to the competition peer review.
a
Missing data n = 56 peer reviewers.
b
Missing data n = 23 peer reviewers.
c
Missing data n = 51 peer reviewers.
d
Missing data n = 34 peer reviewers.
At most, one in 20 peer reviewers was rated as not performing adequately with respect to review quality, participation, or responsiveness (Fig. 3; Table 7). Among those whose performance was judged as below expectations, the overwhelming majority were flagged by Peer Review Committee leaders, but did not receive additional training and (or) mentoring (Table S3). Instead, their performance in the next peer review competition was monitored by CIHR staff. The performance of peer reviewers by sex, career stage, university affiliation, and geographic location is summarised in Tables 8–11. Female and male peer review panel members were judged to have performed similarly (Table 8). Fewer early-career researchers than mid- and senior-career peer review panel members were judged as having potential as a future Chair, Scientific Officer, or Peer Review Mentor (Table 9).
Fig. 3.
Table 7.
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
|---|---|---|---|---|
| n = 991a | n = 1123b | n = 1230c | n = 1094d | |
| Review quality | ||||
| Lacks robustness, n (%) | 53 (5.3) | 59 (5.3) | 59 (4.8) | 46 (4.2) |
| Lacks appropriateness, n (%) | 1(0.1) | 6 (0.5) | 19 (1.5) | 16 (1.5) |
| Participation | ||||
| Low participation, n (%) | 23 (2.3) | 23 (2.0) | 19 (1.5) | 9 (0.8) |
| Major presentation weakness, n (%) | 27 (2.7) | 20 (1.8) | 14 (1.1) | 8 (0.7) |
| Difficult to chair, n (%) | 9 (0.9) | 24 (2.1) | 11 (0.9) | 9 (0.8) |
| Lacks professionalism, n (%) | 14 (1.4) | 8 (0.7) | 9 (0.7) | 6 (0.5) |
| Responsiveness | ||||
| Late submitting review, n (%) | 24 (2.4) | 22 (2.0) | 3 (0.2) | 2 (0.2) |
| Follow-up to submit scores/review, n (%) | 28 (2.8) | 67 (6.0) | 49 (4.0) | 56 (5.1) |
| Follow-up for COI declaration, n (%) | 2 (0.2) | 9 (0.8) | 13 (1.1) | 16 (1.5) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
a
Missing data n = 56 peer reviewers.
b
Missing data n = 23 peer reviewers.
c
Missing data n = 51 peer reviewers.
d
Missing data n = 34 peer reviewers.
Table 8.
| Female peer review panel members | Male peer review panel members | |||||||
|---|---|---|---|---|---|---|---|---|
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
| n = 408a | n = 485b | n = 511c | n = 424d | n = 583a | n = 638e | n = 715f | n = 665d | |
| Performance | ||||||||
| Undertook additional tasks, n (%) | 2 (<1) | 4 (<1) | 10 (2) | 13 (3) | 19 (3) | 7 (1) | 9 (1) | 5 (<1) |
| Discussed additional applications, n (%) | 40 (10) | 45 (9) | 65 (13) | 53 (13) | 81 (14)) | 92 (14) | 91 (13) | 89 (13) |
| Outstanding review, n (%) | 38 (9) | 46 (9) | 62 (12) | 52 (12) | 71 (12) | 93 (15) | 91 (13) | 89 (13) |
| Future potential | ||||||||
| Panel Chair, n (%) | 43 (11) | 78 (16) | 60 (12) | 43 (10) | 93 (16) | 94 (15) | 81 (11) | 69 (10) |
| Panel Scientific Officer, n (%) | 57 (14) | 85 (18) | 78 (15) | 71 (17) | 114 (20) | 111 (17) | 93 (13) | 100 (15) |
| Potential Peer Reviewer Mentor, n (%) | 53 (13) | 87 (18) | 106 (21) | 62 (15) | 92 (16) | 122 (19) | 130 (18) | 98 (15) |
| Review quality | ||||||||
| Lacks robustness, n (%) | 23 (6) | 24 (5) | 18 (4) | 11 (3) | 30 (5) | 35 (5) | 41 (6) | 35 (5) |
| Lacks appropriateness, n (%) | 1 (<1)) | 0 | 5 (1) | 3 (1) | 0 | 6 (1) | 14 (2) | 13 (2) |
| Participation | ||||||||
| Low participation, n (%) | 8 (2) | 9 (2) | 7 (1) | 3 (1) | 15 (3) | 14 (2) | 12 (2) | 6 (1) |
| Major presentation weakness, n (%) | 11 (3) | 13 (3) | 7 (1) | 1 (<1) | 16 (3) | 7 (1) | 7 (1) | 7 (1) |
| Difficult to chair, n (%) | 3 (1) | 8 (2) | 5 (1) | 4 (1) | 6 (1) | 16 (3) | 6 (1) | 5 (1) |
| Lacks professionalism, n (%) | 4 (1) | 4 (1) | 2 (<1) | 3 (1) | 10 (2) | 4 (1) | 7 (1) | 7 (1) |
| Responsiveness | ||||||||
| Late submitting review, n (%) | 10 (2) | 11 (2) | 2 (<1) | 0 | 14 (2) | 11 (2) | 1 (<1) | 2 (<1) |
| Follow-up to submit scores/review, n (%) | 9 (2) | 32 (7) | 19 (4) | 19 (4) | 19 (3) | 35 (5) | 30 (4) | 37 (6) |
| Follow-up for COI declaration, n (%) | 1 (<1) | 4 (1) | 2 (<1) | 7 (2) | 1 (<1) | 5 (1) | 11 (2) | 9 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
a
Missing data from 28 reviewers.
b
Missing data from 12 reviewers.
c
Missing data from 20 reviewers.
d
Missing data from 17 reviewers.
e
Missing data from 11 reviewers.
f
Missing data from 30 reviewers.
Table 9.
| Early-career peer review panel members | Mid-career panel members | Senior-career peer review panel members | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
| n = 120a | n = 142b | n = 132c | n = 140d | n = 398e | n = 460f | n = 494g | n = 421h | n = 426i | n = 476j | n = 535k | n = 458l | |
| Performance | ||||||||||||
| Undertook additional tasks, n (%) | 3 (3) | 3 (2) | 3 (2) | 1 (<1) | 8 (2) | 3 (<1) | 7 (1) | 7 (2) | 10 (2) | 8 (2) | 8 (1) | 8 (1) |
| Discussed additional applications, n (%) | 13 (11) | 12 (8) | 18 (14) | 17 (12) | 55 (14) | 57 (12) | 58 (12) | 55 (13) | 52 (12) | 61 (13) | 73 (14) | 62 (14) |
| Outstanding review, n (%) | 15 (13) | 12 (8) | 17 (13) | 17 (12) | 46 (12) | 57 (12) | 56 (11) | 54 (13) | 48 (11) | 63 (13) | 73 (14) | 62 (14) |
| Future potential | ||||||||||||
| Panel Chair, n (%) | 7 (6) | 6 (4) | 10 (7) | 8 (6) | 43 (11) | 62 (13) | 42 (9) | 44 (10) | 84 (20) | 95 (20) | 84 (16) | 58 (13) |
| Panel Scientific Officer, n (%) | 15 (13) | 17 (12) | 17 (11) | 19 (14) | 59 (15) | 89 (19) | 70 (14) | 74 (18) | 94 (22) | 84 (18) | 78 (15) | 74 (16) |
| Potential Peer Reviewer Mentor, n (%) | 16 (13) | 20 (14) | 16 (12) | 17 (12) | 46 (12) | 84 (18) | 84 (17) | 71 (17) | 81 (19) | 95 (20) | 129 (24) | 67 (15) |
| Review quality | ||||||||||||
| Lacks robustness, n (%) | 7 (6) | 6 (4) | 7 (5) | 5 (4) | 21 (5) | 26 (6) | 22 (4) | 16 (4) | 23 (5) | 25 (5) | 28 (5) | 25 (5) |
| Lacks appropriateness, n (%) | 0 | 0 | 3 (3) | 4 (3) | 0 | 2 (<1) | 8 (2) | 5 (1) | 1 (<1) | 4 (<1) | 8 (1) | 6 (1) |
| Participation | ||||||||||||
| Low participation, n (%) | 4 (3) | 3 (2) | 3 (2) | 0 | 10 (3) | 4 (<1) | 8 (2) | 5 (1) | 9 (2) | 16 (3) | 7 (1) | 4 (<1) |
| Major presentation weakness, n (%) | 1 (<1) | 2 (1) | 1 (<1) | 0 | 12 (3) | 9 (2) | 5 (1) | 4 (1) | 11 (3) | 8 (2) | 8 (1) | 4 (<1) |
| Difficult to chair, n (%) | 3 (3) | 2 (1) | 1 (<1) | 0 | 3 (<1) | 10 (2) | 3 (<1) | 4 (1) | 3 (<1) | 11 (2) | 7 (1) | 4 (<1) |
| Lacks professionalism, n (%) | 2 (2) | 1 (<1) | 0 | 0 | 2 (<1) | 1 (<1) | 6 (1) | 2 (<1) | 10 (1) | 5 (1) | 3 (<1) | 4 (<1) |
| Responsiveness | ||||||||||||
| Late submitting review, n (%) | 2 (2) | 2 (1) | 1 (<1) | 0 | 7 (2) | 8 (2) | 1 (<1) | 0 | 13 (3) | 12 (3) | 1 (<1) | 2 (<1) |
| Follow-up to submit scores/review, n (%) | 1 (<1) | 10 (7) | 3 (2) | 7 (5) | 13 (3) | 26 (6) | 15 (3) | 20 () | 11 (3) | 28 (6) | 27 (5) | 26 (6) |
| Follow-up for COI declaration, n (%) | 0 | 1 (<1) | 1 (<1) | 5 (4) | 0 | 1 (<1) | 5 (1) | 4 (1) | 2 (<1) | 7 (1) | 6 (1) | 5 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
a
Missing data from 3 reviewers.
b
Missing data from 1 reviewer.
c
Missing data from 5 reviewers.
d
Missing data from 4 reviewers.
e
Missing data from 23 reviewers.
f
Missing data from 11 reviewers.
g
Missing data from 17 reviewers.
h
Missing data from 10 reviewers.
i
Missing data from 24 reviewers.
j
Missing data from 10 reviewers.
k
Missing data from 26 reviewers.
l
Missing data from 18 reviewers.
Table 10.
| U15 Group peer review panel members | Non-U15 Group peer review panel members | |||||||
|---|---|---|---|---|---|---|---|---|
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
| n = 834a | n = 960b | n = 1044c | n = 918d | n = 157e | n = 163f | n = 186g | n = 176h | |
| Performance | ||||||||
| Undertook additional tasks, n (%) | 21 (3) | 10 (1) | 16 (2) | 14 (2) | 0 | 1 (1) | 3 (2) | 4 (2) |
| Discussed additional applications, n (%) | 111 (13) | 113 (12) | 138 (13) | 120 (13) | 10 (6) | 24 (15) | 18 (11) | 23 (13) |
| Outstanding review, n (%) | 103 (12) | 115 (12) | 135 (13) | 119 (13) | 6 (4) | 24 (15) | 18 (11) | 23 (13) |
| Future potential | ||||||||
| Panel Chair, n (%) | 121 (15) | 143 (15) | 123 (12) | 96 (10) | 15 (10) | 29 (18) | 18 (11) | 16 (9) |
| Panel Scientific Officer, n (%) | 146 (18) | 173 (18) | 149 (14) | 149 (16) | 25 (16) | 23 (18) | 22 (12) | 22 (13) |
| Potential Peer Reviewer Mentor, n (%) | 125 (15) | 183 (19) | 196 (19) | 136 (15) | 20 (13) | 26 (16) | 40 (22) | 24 (14) |
| Review quality | ||||||||
| Lacks robustness, n (%) | 47 (6) | 44 (5) | 56 (5) | 38 (4) | 6 (4) | 15 (9) | 3 (2) | 8 (5) |
| Lacks appropriateness, n (%) | 1 (<1) | 4 (<1) | 19 (2) | 13 (1) | 0 | 2 (1) | 0 | 3 (2) |
| Participation | ||||||||
| Low participation, n (%) | 19 (2) | 17 (2) | 19 (2) | 7 (1) | 4 (3) | 6 (4) | 0 | 2 (1) |
| Major presentation weakness, n (%) | 23 (3) | 18 (2) | 14 (1) | 5 (1) | 4 (3) | 2 (1) | 0 | 3 (2) |
| Difficult to chair, n (%) | 9 (1) | 22 (2) | 6 (1) | 8 (1) | 0 | 2 (1) | 5 (3) | 1 (1) |
| Lacks professionalism, n (%) | 13 (2) | 6 (1) | 8 (1) | 3 (<1) | 1 (1) | 2 (1) | 1 (1) | 3 (2) |
| Responsiveness | ||||||||
| Late submitting review, n (%) | 20 (2) | 17 (2) | 2 (<1) | 2 (<1) | 4 (3) | 5 (3) | 1 (1) | 0 |
| Follow-up to submit scores/review, n (%) | 24 (3) | 63 (7) | 45 (4) | 49 (5) | 4 (3) | 4 (2) | 4 (2) | 7 (4) |
| Follow-up for COI declaration, n (%) | 2 (<1) | 8 (1) | 12 (1) | 14 (2) | 0 | 1 (1) | 1 (1) | 2 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
a
Missing data from 38 reviewers.
b
Missing data from 20 reviewers.
c
Missing data from 42 reviewers.
d
Missing data from 28 reviewers.
e
Missing data from 18 reviewers.
f
Missing data from 3 reviewers.
g
Missing data from 9 reviewers.
h
Missing data from 2 reviewers.
Table 11.
| Canada peer review panel members | International peer review panel members | |||||||
|---|---|---|---|---|---|---|---|---|
| Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
| n = 968a | n = 1098b | n = 1200c | n = 1076d | n = 23e | n = 25 | n = 30f | n = 18 | |
| Performance | ||||||||
| Undertook additional tasks, n (%) | 21 (2) | 11 (1) | 18 (2) | 18 (2) | 0 | 0 | 1 (2) | 0 |
| Discussed additional applications, n (%) | 117 (12) | 134 (12) | 153 (13) | 141 (13) | 4 (17) | 3 (12) | 3 (10) | 2 (7) |
| Outstanding review, n (%) | 107 (11) | 136 (12) | 151 (13) | 140 (13) | 2 (9) | 3 (12) | 2 (7) | 2 (7) |
| Future potential | ||||||||
| Panel Chair, n (%) | 134 (14) | 167 (15) | 140 (12) | 112 (10) | 2 (9) | 5 (20) | 1 (3) | 0 |
| Panel Scientific Officer, n (%) | 169 (17) | 193 (16) | 170 (14) | 171 (16) | 2 (9) | 3 (12) | 1 (3) | 0 |
| Potential Peer Reviewer Mentor, n (%) | 143 (15) | 207 (19) | 234 (20) | 160 (15) | 2 (9) | 2 (8) | 2 (7) | 0 |
| Review quality | ||||||||
| Lacks robustness, n (%) | 53 (5) | 55 (5) | 58 (5) | 45 (4) | 0 | 4 (16) | 1 (3) | 1 (3) |
| Lacks appropriateness, n (%) | 1 (<1) | 5 (1) | 19 (2) | 16 (2) | 0 | 1 (4) | 0 | 0 |
| Participation | ||||||||
| Low participation, n (%) | 23 (2) | 23 (2) | 19 (2) | 9 (1) | 0 | 0 | 0 | 0 |
| Major presentation weakness, n (%) | 27 (3) | 20 (2) | 14 (1) | 7 (1) | 0 | 0 | 0 | 1 (3) |
| Difficult to chair, n (%) | 9 (1) | 23 (2) | 11 (1) | 9 (1) | 0 | 1 (4) | 0 | 0 |
| Lacks professionalism, n (%) | 14 (1) | 7 (1) | 9 (1) | 5 (1) | 0 | 1 (4) | 0 | 1 (3) |
| Responsiveness | ||||||||
| Late submitting review, n (%) | 24 (2) | 20 (2) | 3 (<1) | 2 (<1) | 0 | 2 (8) | 0 | 0 |
| Follow-up to submit scores/review, n (%) | 28 (3) | 65 (6) | 48 (4) | 54 (5) | 0 | 2 (8) | 1 (3) | 2 (7) |
| Follow-up for COI declaration, n (%) | 2 (<1) | 9 (1) | 12 (1) | 16 (2) | 0 | 0 | 1 (3) | 0 |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
a
Missing data from 50 reviewers.
b
Missing data from 23 reviewers.
c
Missing data from 46 reviewers.
d
Missing data from 33 reviewers.
e
Missing data from 6 reviewers.
f
Missing data from 5 reviewers.
g
Missing data from 1 reviewer.
Discussion
We contributed to the limited literature in the important field of grant peer review in at least three ways. First, we highlighted the utility of one tool to measure grant peer review quality (CIHR's RQA tool). There is a dearth of tools to measure the quality of peer review in journals (Superchi et al. 2019) and more so in the less-researched field of grant allocation. The RQA tool was developed in 2017, pilot-tested, and implemented in CIHR's flagship health research funding program—the Project Grant Competition—from 2019. We encourage other funders to consider how they evaluate peer review quality, and to collaborate to develop standard tools that are tailored to the specific context of grant peer review.
Our second contribution, the data from over 4000 contributions to peer review, suggests that Peer Review Committee leaders—Chairs and Scientific Officers who applied the RQA tool—feel the Canadian project grant peer review process is in relatively good health. These leaders reported very few cases where reviewers failed to meet an acceptable standard. In 3 years (four grant competitions), peer reviewers adjudicated approximately 8000 grant applications and awarded approximately C$1.5 billion. These are important data to share because without measuring, one can only speculate. Ongoing monitoring in the CIHR Project Grant Competition, and in collaboration with other health research funders, will establish the value of monitoring grant peer review and help to embed a culture of quality peer review practice underpinned by continuous monitoring and improvement.
There are 14 domains in the RQA tool, which broadly measure “negatives” and “positives” related to the appropriateness, robustness, and utility of the peer reviewer's feedback, the reviewer's potential to contribute to leadership roles in the peer review committee, and the reviewer's participation in the peer review process. We speculate that CIHR's over 15 years of experience in this type of grant peer review (committees of approximately 20 experts who do not send applications out for peer review) has contributed to the current relatively good health of the system. Individuals who may have scored poorly on the RQA in the past are likely to have been removed from the pool of experts who make up the approximately 60 peer review committees. We note that the RQA only captures certain domains of grant peer review and other types of research (qualitative research, including ethnographic approaches and interviews) would complement evaluation of quality grant peer review.
A third element of our research will interest funders who are exploring whether grant peer review can be done “effectively” via platforms such as Microsoft Teams or Zoom. We qualify the word “effectively” as we appreciate it is difficult to define, and the debate over “in-person” versus remote (i.e., via videoconference) for various types of meetings is charged. We enter the debate respectfully, only to share RQA data from relevant years (2019–2021), which included the COVID-19 pandemic.
One of the CIHR Project Competitions we report data for took place pre-COVID (face-to-face meetings) and the other three were held using Microsoft Teams. The Peer Review Committee members were under well-documented strain during the last three peer review competitions (re-deployment to clinical roles, working from home, major changes to child care/school arrangements, etc.). Despite this, RQA ratings, which reflected more than 2000 individual Peer Review Committee members’ work, remained high.
Research on research in grant peer review
Research in the grant peer review field has distinguished process questions from questions about how scientific merit is defined (Gallo et al. 2020; Pina et al. 2021). One key paper suggests that a grant peer review committee process, with discussion, alters about one in 10 grant rankings that come directly from external peer review (Carpenter et al. 2015). That is, 90% of the grants that external peer reviewers would fund if they were the final arbiters are funded after peer review committee discussion. Whether the CIHR-funded grants (about 15%–20%) during the 3-year study period were the “best” grants by any external measure of excellence warrants further research.
Funders continue to explore ways to bring greater equity to how research funding is allocated (Bendiscioli and Garfinkel 2021; Hill 2021; Horbach et al. 2022; Recio-Saucedo et al. 2022). It is appropriate to embed rigorous evaluation of the outcomes of different approaches to funding decisions (Shepherd et al. 2018), and to commit to studying the perceptions of those involved in the decisions (including applicants, peer reviewers, and funders) among others (Liu et al. 2020; Steiner Davis et al. 2020). Given the seemingly crucial role of the Peer Review Committee leaders (Gallo et al. 2020), CIHR has commenced work to develop a Quality Feedback form to assess their performance.
Limitations
The binary response options in the Reviewer Quality Feedback form mean we could not capture nuance in reviewers’ performance. The Reviewer Quality Feedback form was developed specifically for the RQA program, and has not yet undergone clinimetric assessment. Therefore, its inter-rater reliability (among other measurement properties) is unknown. Although explanatory text accompanies each item on the Reviewer Quality Feedback form, it is likely that different peer review leaders interpreted the Reviewer Quality Feedback form items in different ways.
The Reviewer Quality Feedback form was developed to monitor reviewers’ performance in a way that was reproducible, carried minimal burden for the people who were tasked with completing the form (i.e., Chairs and Scientific Officers), and so that CIHR staff could analyse and interpret the results in a timely fashion, and then act on the results (as required) as quickly as possible. CIHR plans to review all items in the Reviewer Quality Feedback form after sufficient data to provide an accurate picture of the strengths and limitations of the form are collected.
Given the Peer Review Committee leaders (Chairs and Scientific Officers) completed the Reviewer Quality Feedback forms immediately after the Committee for which they were responsible completed its work, we believe the risk for recall bias is low. It is routine practice for CIHR Program Delivery staff to attend the Peer Review Committee meetings and provide administrative support during the meetings. The CIHR staff who attended the Peer Review Committee meetings also independently validated each completed Reviewer Quality Feedback form.
Conclusion
In the 3-year period studied, which included the COVID-19 pandemic, the quality of peer review for Project Grant Competitions at Canada's national health research funding agency was consistent over four competitions. Ninety-five percent of peer reviewers met the standards expected by peer review leaders for review quality, participation, and responsiveness. Canada's national health research funding agency (CIHR) has a track record of contributing to research on research, and this provides their first output related to grant peer review quality. There is a specific need for more research to define, measure, and monitor quality in grant peer review, and on the processes involved more broadly in allocating health research grant funding.
Acknowledgements
Zakirullah Sanaye (Advisor, CIHR College of Reviewers), Nicole Haddadian (Lead, CIHR College of Reviewers), and Simon Partridge (Senior Advisor, CIHR College of Reviewers) constructed the project database and helped produce the summary data.
References
Bendiscioli S., Garfinkel M. 2021. Dealing with the limits of peer review with innovative approaches to allocating research funding. EMBO Science Policy Programme. Available from https://www.embo.org/documents/science_policy/peer_review_report.pdf [accessed 31 January 2023].
Canadian Institutes of Health Research. 2022a. CIHR Standards of Practice for Peer Review [online]. Available from https://cihr-irsc.gc.ca/e/51645.html [accessed 4 July 2022].
Canadian Institutes of Health Research. 2022b. Conflict of Interest and Confidentiality Agreement for Peer Reviewers and Peer Review Observers [online]. Available from https://cihr-irsc.gc.ca/e/46378.html [accessed 4 July 2022].
Canadian Institutes of Health Research. 2022c. Glossary of Funding-Related Terms: researcher (independent) [online]. Available from https://cihr-irsc.gc.ca/e/34190.html#r6 [accessed 6 July 2022].
Canadian Institutes of Health Research. 2022d. Guide on Handling Documents Used in Peer Review [online]. Available from https://cihr-irsc.gc.ca/e/40261.html [accessed 4 July 2022].
Canadian Institutes of Health Research. 2022e. Review Quality Assurance Process [online]. Available from https://cihr-irsc.gc.ca/e/51644.html [accessed 28 June 2022].
Carpenter A.S., Sullivan J.H., Deshmukh A., Glisson S.R., Gallo S.A. 2015. A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.BMJ Open, 5: e009138.
European Science Foundation. 2011a. ESF Survey Analysis Report on Peer Review Practices. Available from https://www.esf.org/fileadmin/user_upload/esf/PeerReview-Practices_Survey2011.pdf [accessed 31 January 2023].
European Science Foundation. 2011b. European Peer Review Guide: integrating policies and practices into coherent procedures. Available from https://www.esf.org/fileadmin/user_upload/esf/European_Peer_Review_Guide_2011.pdf [accessed 31 January 2023].
European Science Foundation. 2012. Evaluation in research and research funding organisations: European practices. In A report by the ESF Member Organisation Forum on Evaluation of Publicly Funded Research. Available from http://archives.esf.org/coordinating-research/mo-fora/evaluation-of-publicly-funded-research.html [accessed 31 January 2023].
Gallo S.A., Schmaling K.B., Thompson L.A., Glisson S.R. 2020. Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Research Integrity and Peer Review, 5: 7.
Guthrie S., Ghiga I., Wooding S. 2018. What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies. RAND Corporation, Santa Monica, CA.
Hill M. 2021. The jury is out: a new approach to awarding science prizes [version 1; peer review: 1 approved, 1 approved with reservations]. F1000Research, 10: 1237.
Horbach S., Tijdink J.K., Louter L.M. 2022. Partial lottery can make grant allocation more fair, more efficient, and more diverse. Science and Public Policy, scac009.
Liu M., Choy V., Clarke P., Barnett A., Blakely T., Pomeroy L. 2020. The acceptabiility of using a lottery to allocate research funding: a survey of applicants. Research Integrity and Peer Review, 5: 3.
Pina D.G., Buljan I., Hren D., Marušić A. 2021. A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018. Elife, 10: e59338.
Recio-Saucedo A., Crane K., Meadmore K., Fackrell K., Church H., Fraser S., Blatch-Jones A. 2022. What works for peer review and decision-making in research funding: a realist synthesis.Research Integrity and Peer Review, 7: 2.
Shepherd J., Frampton G.K., Pickett K., Wyatt J.C. 2018. Peer review of health research funding proposals: a systematic map and systematic review of innovations for effectiveness and efficiency.PLoS ONE, 13: e0196914.
Steiner Davis M.L.E., Conner T.R., Miller-Bains K., Shapard L. 2020. What makes an effective grants peer reviewer? an exploratory study of the necessary skills. PLoS ONE, 15: e0232327.
Superchi C., González J., Solà I., Cobo E., Hren D., Boutron I. 2019. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Medical Research Methodology, 19: 48.
U15 Group of Canadian Research Universities. 2022. About Us [online]. Available from https://u15.ca/about-us [accessed 26 June 2022].
Supplementary material
Supplementary Material 1 (DOCX / 27.5 KB).
- Download
- 27.50 KB
Information & Authors
Information
Published In
FACETS
Volume 8 • January 2023
Pages: 1 - 14
Editor: Michael John Evans
History
Received: 7 August 2022
Accepted: 17 May 2023
Version of record online: 7 August 2023
Copyright
© 2023 The Crown and authors Ardern, Tamblyn, Moher, and Khan. This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Data Availability Statement
All data reported in this manuscript are available from Ardern, Clare, 2022, “Dataset for the Canadian Institutes of Health Research (CIHR) Research Quality Assurance Program 2019 to 2021,” https://doi.org/10.5683/SP3/KZL1M1, Borealis, V1, UNF:6:jL/k/v6xO9XWhSqMqe2WfA==[fileUNF].
Key Words
Sections
Subjects
Plain Language Summary
Performance of Canadian Institutes of Health Research Peer Reviewers When Assessing Grant Applications Submitted From 2019 to 2021
Authors
Author Contributions
Conceptualization: CLA, NM, SN, RT, DM, AM, KMK
Data curation: CLA, NM, SN
Formal analysis: CLA
Funding acquisition: KMK
Investigation: CLA
Methodology: CLA, RT, DM
Project administration: CLA, NM, SN
Supervision: CLA, NM, SN
Validation: NM, SN
Visualization: CLA, KMK
Writing – original draft: CLA, KMK
Writing – review and editing: CLA, NM, SN, RT, DM, AM, KMK
Competing Interests
DM is a Subject Editor for FACETS. NM, SN and AM are Canadian Government employees (of CIHR) who contributed to this manuscript in the course of their regular employment. AM is responsible for co-ordinating the delivery of peer review for CIHR. KMK is Scientific Director for the CIHR Institute of Musculoskeletal Health and Arthritis. CLA and RT declare they have no competing interests.
Funding Information
This work was supported by a CIHR Research Operating Grant (Scientific Directors) held by KMK. The funder (CIHR) had no role in design and conduct of the study, CIHR's College of Reviewers coordinated data collection, management, and analysis as part of its mandate to foster and deliver high-quality peer review for health research in Canada. CIHR did not participate in interpreting the data or preparing the manuscript; CIHR reviewed and approved the manuscript prior to submission. CIHR did not participate in the decision to submit the manuscript for publication.
Metrics & Citations
Metrics
Other Metrics
Citations
Cite As
Clare L. Ardern, Nadia Martino, Sammy Nag, Robyn Tamblyn, David Moher, Adrian Mota, and Karim M. Khan. 2023. Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian Institutes of Health Research Project Grant Competition. FACETS.
8(): 1-14. https://doi.org/10.1139/facets-2022-0175
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
1. Biomedical research grant resubmission: rates and factors related to success – a scoping review