Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian Institutes of Health Research Project Grant Competition
Abstract
Introduction
Context
Review Committee preparation and training
Peer review process
Methods
CIHR's Quality Assurance Program
Effective contributor | Outstanding contributor | Needs improvement | Other considerations | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Y/N | Review quality | Review quality | Participation | Review quality | Participation | ||||||
Y/N | Participation | Y/N | Reviews went over and above expectations (e.g., insightful comments—detailing both strengths and weaknesses—that will significantly aid both the applicant and other reviewers) | Y/N | Constructively participated in application discussions NOT assigned to them | Y/N | Review(s) lack appropriateness (e.g., contain biased and (or) inappropriate comments) | Y/N | Lacks professionalism (e.g., stubborn, combative, overly critical/vocal, and inappropriate/biased comments) | Y/N | Demonstrates potential as future Chair |
Y/N | Agreed to take on additional tasks on short notice (e.g., to review more applications) | Y/N | Review(s) lacks robustness (e.g., contain insufficient details to justify given ratings) | Y/N | Difficult to chair (e.g., not responsive to Chair's suggestions, interrupts, and speaks too much) | Y/N | Demonstrates potential as future Scientific Officer | ||||
Y/N | Review(s) lack utility (e.g., comments are not constructive and not helpful to applicants to improve their future submissions) | Y/N | Major presentation weaknesses (e.g., not well prepared, difficulty presenting, difficult to hear, and unclear/unfocused comments) | Y/N | Demonstrates potential as future Peer Reviewer Mentor | ||||||
Y/N | Low participation level (e.g., not engaged as primary reviewer in discussions and absentminded) |
Note: Peer reviewers who recuse themselves from a committee discussion for conflicts of interest are not assessed.
Developing the Research Quality Assurance Feedback form
Assessing the quality of the peer review
Review quality criterion | Definition | Interpretation |
---|---|---|
Appropriateness | Review comments are fair, understandable, confidential, and respectful | • Review respects CIHR's Conflict of Interest and Confidentiality policy • Absence of comments that suggest bias against the applicant(s) due to sex, ethnicity, age, language, career stage, institutional affiliation, or geographic location • Review is original, and written in clear and understandable language • Absence of comments that can be construed as sarcastic, flippant, or arrogant |
Robustness | Review is thorough, complete, and credible | • Review contains a detailed justification of each rating, including meaningful and clearly expressed descriptions of both the application strengths and weaknesses • Comments align with the given rating • Review addresses all applicable adjudication criteria and does not include information that is not relevant to the adjudication criteria • All comments on grant content are factually correct • Absence of statements that could call into question the reviewer's scientific knowledge or expertise |
Utility | Review provides feedback that addresses the needs of reviewers, applicants, and funders | • Review comments are constructive and may help applicants to improve their future submissions and (or) advance their research • Review contains information that allows other reviewers to understand the reviewer's rating(s) • Review is detailed enough to be used by CIHR to evaluate and refine review process elements |
Participants
Peer review committee member characteristics
Process for obtaining consent
Procedures
Data analysis and synthesis
Results
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
---|---|---|---|---|
Chairs and Scientific Officers (n) | ||||
Participants | 167 | 174 | 181 | 172 |
Provided feedback for individual reviewers | 108 | 152 | 158 | 154 |
Provided feedback for committee only | 0 | 0 | 4 | 6 |
No feedback provided | 59 | 22 | 19 | 12 |
Peer Review Committees (n) | ||||
Total committees | 59 | 57 | 61 | 57 |
Committees with complete RQA data | 59 | 57 | 61 | 57 |
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
---|---|---|---|---|
n = 108 | n = 152 | n = 158 | n = 154 | |
Sex (n, %) | ||||
Female | 48 (44) | 78 (51) | 84 (53) | 86 (56) |
Male | 60 (56) | 74 (49) | 74 (47) | 68 (44) |
Career stage (n, %) | ||||
Early | 3 (3) | 1 (<1) | 1 (<1) | 5 (3) |
Mid | 36 (33) | 61 (40) | 47 (30) | 58 (38) |
Senior | 66 (61) | 84 (55) | 102 (65) | 87 (56) |
Missing | 3 (3) | 6 (4) | 8 (5) | 4 (3) |
Affiliation (n, %) | ||||
U15 | 81 (75) | 113 (74) | 113 (72) | 114 (74) |
Non-U15 | 17 (16) | 23 (15) | 24 (15) | 22 (14) |
Missing | 10 (9) | 16 (11) | 21 (13) | 18 (12) |
CIHR College of Reviewers member (n, %) | ||||
Yes | 101 (94) | 125 (82) | 132 (84) | 133 (86) |
No | 7 (6) | 27 (18) | 26 (16) | 21 (14) |
Note: U15, institution is a member of the U15 Group of Canadian Research Universities.
Peer review committee composition
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
---|---|---|---|---|
n = 991 | n = 1123 | n = 1230 | n = 1094 | |
Sex (n, %) | ||||
Female | 408 (41) | 485 (43) | 511 (42) | 424 (39) |
Male | 583 (59) | 638 (57) | 715 (58) | 665 (61) |
Missing | 0 | 0 | 4 (<1) | 5 (<1) |
Career stage (n, %) | ||||
Early | 59 (6) | 99 (9) | 132 (11) | 140 (13) |
Mid | 384 (39) | 456 (41) | 494 (41) | 421 (38) |
Senior | 502 (51) | 523 (47) | 535 (43) | 458 (42) |
Missing | 46 (5) | 45 (4) | 69 (6) | 75 (7) |
Affiliation (n, %) | ||||
U15 | 834 (84) | 960 (85) | 1044 (85) | 918 (84) |
Non-U15 | 157 (16) | 163 (15) | 186 (15) | 176 (16) |
CIHR College of Reviewers member (n, %) | ||||
Yes | 837 (84) | 969 (86) | 1008 (82) | 872 (80) |
No | 102 (10) | 141 (13) | 205 (17) | 222 (20) |
Missing | 52 (5) | 13 (1) | 17 (1) | 0 |
Location | ||||
Canada | 968 (98) | 1098 (98) | 1200 (98) | 1076 (98) |
International (n, %) | 23 (2) | 25 (2) | 30 (2) | 18 (2) |
Project grant applications assigned^ (n) | ||||
Min | 1 | 1 | 1 | 1 |
Max | 11 | 10 | 10 | 10 |
Median | 7 | 7 | 6 | 6 |
Competitions serving as peer reviewer* (n) | ||||
Min | 0 | 0 | 0 | 0 |
Max | 3 | 4 | 3 | 3 |
Median | 1 | 1 | 1 | 1 |
Grant funding history as Principal Investigator∼ (n) | ||||
Min | 0 | 0 | 0 | 0 |
Max | 7 | 8 | 8 | 8 |
Median | 2 | 2 | 2 | 2 |
Note: U15, institution is a member of the U15 Group of Canadian Research Universities.
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
---|---|---|---|---|
n = 991a | n = 1123b | n = 1230c | n = 1094d | |
Performance | ||||
Undertook additional tasks, n (%) | 51 (5.1) | 23 (2.0) | 47 (3.8) | 39 (3.6) |
Discussed additional applications, n (%) | 309 (31.2) | 430 (38.3) | 380 (30.9) | 311 (28.4) |
Outstanding review, n (%) | 285 (28.8) | 487 (43.4) | 386 (31.4) | 356 (32.5) |
Future potential | ||||
Committee Chair, n (%) | 136 (13.7) | 172 (15.3) | 141 (11.5) | 112 (10.2) |
Committee Scientific Officer, n (%) | 171 (17.3) | 196 (17.5) | 171 (13.9) | 171 (15.6) |
Potential Peer Reviewer Mentor, n (%) | 145 (14.6) | 209 (18.6) | 236 (19.2) | 160 (14.6) |
Note: N = number of peer reviewers who contributed to the competition peer review.
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
---|---|---|---|---|
n = 991a | n = 1123b | n = 1230c | n = 1094d | |
Review quality | ||||
Lacks robustness, n (%) | 53 (5.3) | 59 (5.3) | 59 (4.8) | 46 (4.2) |
Lacks appropriateness, n (%) | 1(0.1) | 6 (0.5) | 19 (1.5) | 16 (1.5) |
Participation | ||||
Low participation, n (%) | 23 (2.3) | 23 (2.0) | 19 (1.5) | 9 (0.8) |
Major presentation weakness, n (%) | 27 (2.7) | 20 (1.8) | 14 (1.1) | 8 (0.7) |
Difficult to chair, n (%) | 9 (0.9) | 24 (2.1) | 11 (0.9) | 9 (0.8) |
Lacks professionalism, n (%) | 14 (1.4) | 8 (0.7) | 9 (0.7) | 6 (0.5) |
Responsiveness | ||||
Late submitting review, n (%) | 24 (2.4) | 22 (2.0) | 3 (0.2) | 2 (0.2) |
Follow-up to submit scores/review, n (%) | 28 (2.8) | 67 (6.0) | 49 (4.0) | 56 (5.1) |
Follow-up for COI declaration, n (%) | 2 (0.2) | 9 (0.8) | 13 (1.1) | 16 (1.5) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
Female peer review panel members | Male peer review panel members | |||||||
---|---|---|---|---|---|---|---|---|
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
n = 408a | n = 485b | n = 511c | n = 424d | n = 583a | n = 638e | n = 715f | n = 665d | |
Performance | ||||||||
Undertook additional tasks, n (%) | 2 (<1) | 4 (<1) | 10 (2) | 13 (3) | 19 (3) | 7 (1) | 9 (1) | 5 (<1) |
Discussed additional applications, n (%) | 40 (10) | 45 (9) | 65 (13) | 53 (13) | 81 (14)) | 92 (14) | 91 (13) | 89 (13) |
Outstanding review, n (%) | 38 (9) | 46 (9) | 62 (12) | 52 (12) | 71 (12) | 93 (15) | 91 (13) | 89 (13) |
Future potential | ||||||||
Panel Chair, n (%) | 43 (11) | 78 (16) | 60 (12) | 43 (10) | 93 (16) | 94 (15) | 81 (11) | 69 (10) |
Panel Scientific Officer, n (%) | 57 (14) | 85 (18) | 78 (15) | 71 (17) | 114 (20) | 111 (17) | 93 (13) | 100 (15) |
Potential Peer Reviewer Mentor, n (%) | 53 (13) | 87 (18) | 106 (21) | 62 (15) | 92 (16) | 122 (19) | 130 (18) | 98 (15) |
Review quality | ||||||||
Lacks robustness, n (%) | 23 (6) | 24 (5) | 18 (4) | 11 (3) | 30 (5) | 35 (5) | 41 (6) | 35 (5) |
Lacks appropriateness, n (%) | 1 (<1)) | 0 | 5 (1) | 3 (1) | 0 | 6 (1) | 14 (2) | 13 (2) |
Participation | ||||||||
Low participation, n (%) | 8 (2) | 9 (2) | 7 (1) | 3 (1) | 15 (3) | 14 (2) | 12 (2) | 6 (1) |
Major presentation weakness, n (%) | 11 (3) | 13 (3) | 7 (1) | 1 (<1) | 16 (3) | 7 (1) | 7 (1) | 7 (1) |
Difficult to chair, n (%) | 3 (1) | 8 (2) | 5 (1) | 4 (1) | 6 (1) | 16 (3) | 6 (1) | 5 (1) |
Lacks professionalism, n (%) | 4 (1) | 4 (1) | 2 (<1) | 3 (1) | 10 (2) | 4 (1) | 7 (1) | 7 (1) |
Responsiveness | ||||||||
Late submitting review, n (%) | 10 (2) | 11 (2) | 2 (<1) | 0 | 14 (2) | 11 (2) | 1 (<1) | 2 (<1) |
Follow-up to submit scores/review, n (%) | 9 (2) | 32 (7) | 19 (4) | 19 (4) | 19 (3) | 35 (5) | 30 (4) | 37 (6) |
Follow-up for COI declaration, n (%) | 1 (<1) | 4 (1) | 2 (<1) | 7 (2) | 1 (<1) | 5 (1) | 11 (2) | 9 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
Early-career peer review panel members | Mid-career panel members | Senior-career peer review panel members | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
n = 120a | n = 142b | n = 132c | n = 140d | n = 398e | n = 460f | n = 494g | n = 421h | n = 426i | n = 476j | n = 535k | n = 458l | |
Performance | ||||||||||||
Undertook additional tasks, n (%) | 3 (3) | 3 (2) | 3 (2) | 1 (<1) | 8 (2) | 3 (<1) | 7 (1) | 7 (2) | 10 (2) | 8 (2) | 8 (1) | 8 (1) |
Discussed additional applications, n (%) | 13 (11) | 12 (8) | 18 (14) | 17 (12) | 55 (14) | 57 (12) | 58 (12) | 55 (13) | 52 (12) | 61 (13) | 73 (14) | 62 (14) |
Outstanding review, n (%) | 15 (13) | 12 (8) | 17 (13) | 17 (12) | 46 (12) | 57 (12) | 56 (11) | 54 (13) | 48 (11) | 63 (13) | 73 (14) | 62 (14) |
Future potential | ||||||||||||
Panel Chair, n (%) | 7 (6) | 6 (4) | 10 (7) | 8 (6) | 43 (11) | 62 (13) | 42 (9) | 44 (10) | 84 (20) | 95 (20) | 84 (16) | 58 (13) |
Panel Scientific Officer, n (%) | 15 (13) | 17 (12) | 17 (11) | 19 (14) | 59 (15) | 89 (19) | 70 (14) | 74 (18) | 94 (22) | 84 (18) | 78 (15) | 74 (16) |
Potential Peer Reviewer Mentor, n (%) | 16 (13) | 20 (14) | 16 (12) | 17 (12) | 46 (12) | 84 (18) | 84 (17) | 71 (17) | 81 (19) | 95 (20) | 129 (24) | 67 (15) |
Review quality | ||||||||||||
Lacks robustness, n (%) | 7 (6) | 6 (4) | 7 (5) | 5 (4) | 21 (5) | 26 (6) | 22 (4) | 16 (4) | 23 (5) | 25 (5) | 28 (5) | 25 (5) |
Lacks appropriateness, n (%) | 0 | 0 | 3 (3) | 4 (3) | 0 | 2 (<1) | 8 (2) | 5 (1) | 1 (<1) | 4 (<1) | 8 (1) | 6 (1) |
Participation | ||||||||||||
Low participation, n (%) | 4 (3) | 3 (2) | 3 (2) | 0 | 10 (3) | 4 (<1) | 8 (2) | 5 (1) | 9 (2) | 16 (3) | 7 (1) | 4 (<1) |
Major presentation weakness, n (%) | 1 (<1) | 2 (1) | 1 (<1) | 0 | 12 (3) | 9 (2) | 5 (1) | 4 (1) | 11 (3) | 8 (2) | 8 (1) | 4 (<1) |
Difficult to chair, n (%) | 3 (3) | 2 (1) | 1 (<1) | 0 | 3 (<1) | 10 (2) | 3 (<1) | 4 (1) | 3 (<1) | 11 (2) | 7 (1) | 4 (<1) |
Lacks professionalism, n (%) | 2 (2) | 1 (<1) | 0 | 0 | 2 (<1) | 1 (<1) | 6 (1) | 2 (<1) | 10 (1) | 5 (1) | 3 (<1) | 4 (<1) |
Responsiveness | ||||||||||||
Late submitting review, n (%) | 2 (2) | 2 (1) | 1 (<1) | 0 | 7 (2) | 8 (2) | 1 (<1) | 0 | 13 (3) | 12 (3) | 1 (<1) | 2 (<1) |
Follow-up to submit scores/review, n (%) | 1 (<1) | 10 (7) | 3 (2) | 7 (5) | 13 (3) | 26 (6) | 15 (3) | 20 () | 11 (3) | 28 (6) | 27 (5) | 26 (6) |
Follow-up for COI declaration, n (%) | 0 | 1 (<1) | 1 (<1) | 5 (4) | 0 | 1 (<1) | 5 (1) | 4 (1) | 2 (<1) | 7 (1) | 6 (1) | 5 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
U15 Group peer review panel members | Non-U15 Group peer review panel members | |||||||
---|---|---|---|---|---|---|---|---|
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
n = 834a | n = 960b | n = 1044c | n = 918d | n = 157e | n = 163f | n = 186g | n = 176h | |
Performance | ||||||||
Undertook additional tasks, n (%) | 21 (3) | 10 (1) | 16 (2) | 14 (2) | 0 | 1 (1) | 3 (2) | 4 (2) |
Discussed additional applications, n (%) | 111 (13) | 113 (12) | 138 (13) | 120 (13) | 10 (6) | 24 (15) | 18 (11) | 23 (13) |
Outstanding review, n (%) | 103 (12) | 115 (12) | 135 (13) | 119 (13) | 6 (4) | 24 (15) | 18 (11) | 23 (13) |
Future potential | ||||||||
Panel Chair, n (%) | 121 (15) | 143 (15) | 123 (12) | 96 (10) | 15 (10) | 29 (18) | 18 (11) | 16 (9) |
Panel Scientific Officer, n (%) | 146 (18) | 173 (18) | 149 (14) | 149 (16) | 25 (16) | 23 (18) | 22 (12) | 22 (13) |
Potential Peer Reviewer Mentor, n (%) | 125 (15) | 183 (19) | 196 (19) | 136 (15) | 20 (13) | 26 (16) | 40 (22) | 24 (14) |
Review quality | ||||||||
Lacks robustness, n (%) | 47 (6) | 44 (5) | 56 (5) | 38 (4) | 6 (4) | 15 (9) | 3 (2) | 8 (5) |
Lacks appropriateness, n (%) | 1 (<1) | 4 (<1) | 19 (2) | 13 (1) | 0 | 2 (1) | 0 | 3 (2) |
Participation | ||||||||
Low participation, n (%) | 19 (2) | 17 (2) | 19 (2) | 7 (1) | 4 (3) | 6 (4) | 0 | 2 (1) |
Major presentation weakness, n (%) | 23 (3) | 18 (2) | 14 (1) | 5 (1) | 4 (3) | 2 (1) | 0 | 3 (2) |
Difficult to chair, n (%) | 9 (1) | 22 (2) | 6 (1) | 8 (1) | 0 | 2 (1) | 5 (3) | 1 (1) |
Lacks professionalism, n (%) | 13 (2) | 6 (1) | 8 (1) | 3 (<1) | 1 (1) | 2 (1) | 1 (1) | 3 (2) |
Responsiveness | ||||||||
Late submitting review, n (%) | 20 (2) | 17 (2) | 2 (<1) | 2 (<1) | 4 (3) | 5 (3) | 1 (1) | 0 |
Follow-up to submit scores/review, n (%) | 24 (3) | 63 (7) | 45 (4) | 49 (5) | 4 (3) | 4 (2) | 4 (2) | 7 (4) |
Follow-up for COI declaration, n (%) | 2 (<1) | 8 (1) | 12 (1) | 14 (2) | 0 | 1 (1) | 1 (1) | 2 (1) |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
Canada peer review panel members | International peer review panel members | |||||||
---|---|---|---|---|---|---|---|---|
Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | Fall 2019 | Fall 2020 | Spring 2021 | Fall 2021 | |
n = 968a | n = 1098b | n = 1200c | n = 1076d | n = 23e | n = 25 | n = 30f | n = 18 | |
Performance | ||||||||
Undertook additional tasks, n (%) | 21 (2) | 11 (1) | 18 (2) | 18 (2) | 0 | 0 | 1 (2) | 0 |
Discussed additional applications, n (%) | 117 (12) | 134 (12) | 153 (13) | 141 (13) | 4 (17) | 3 (12) | 3 (10) | 2 (7) |
Outstanding review, n (%) | 107 (11) | 136 (12) | 151 (13) | 140 (13) | 2 (9) | 3 (12) | 2 (7) | 2 (7) |
Future potential | ||||||||
Panel Chair, n (%) | 134 (14) | 167 (15) | 140 (12) | 112 (10) | 2 (9) | 5 (20) | 1 (3) | 0 |
Panel Scientific Officer, n (%) | 169 (17) | 193 (16) | 170 (14) | 171 (16) | 2 (9) | 3 (12) | 1 (3) | 0 |
Potential Peer Reviewer Mentor, n (%) | 143 (15) | 207 (19) | 234 (20) | 160 (15) | 2 (9) | 2 (8) | 2 (7) | 0 |
Review quality | ||||||||
Lacks robustness, n (%) | 53 (5) | 55 (5) | 58 (5) | 45 (4) | 0 | 4 (16) | 1 (3) | 1 (3) |
Lacks appropriateness, n (%) | 1 (<1) | 5 (1) | 19 (2) | 16 (2) | 0 | 1 (4) | 0 | 0 |
Participation | ||||||||
Low participation, n (%) | 23 (2) | 23 (2) | 19 (2) | 9 (1) | 0 | 0 | 0 | 0 |
Major presentation weakness, n (%) | 27 (3) | 20 (2) | 14 (1) | 7 (1) | 0 | 0 | 0 | 1 (3) |
Difficult to chair, n (%) | 9 (1) | 23 (2) | 11 (1) | 9 (1) | 0 | 1 (4) | 0 | 0 |
Lacks professionalism, n (%) | 14 (1) | 7 (1) | 9 (1) | 5 (1) | 0 | 1 (4) | 0 | 1 (3) |
Responsiveness | ||||||||
Late submitting review, n (%) | 24 (2) | 20 (2) | 3 (<1) | 2 (<1) | 0 | 2 (8) | 0 | 0 |
Follow-up to submit scores/review, n (%) | 28 (3) | 65 (6) | 48 (4) | 54 (5) | 0 | 2 (8) | 1 (3) | 2 (7) |
Follow-up for COI declaration, n (%) | 2 (<1) | 9 (1) | 12 (1) | 16 (2) | 0 | 0 | 1 (3) | 0 |
Note: N = number of peer reviewers who contributed to the competition peer review; COI, conflict of interest.
Discussion
Research on research in grant peer review
Limitations
Conclusion
Acknowledgements
References
Supplementary material
- Download
- 27.50 KB
Information & Authors
Information
Published In
History
Copyright
Data Availability Statement
Key Words
Sections
Subjects
Plain Language Summary
Authors
Author Contributions
Competing Interests
Funding Information
Metrics & Citations
Metrics
Other Metrics
Citations
Cite As
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
There are no citations for this item