How to set the agenda for training in responsible conduct of research using the target audience as a narrative guide
Abstract
There is a growing interest in training on responsible conduct of research (RCR). The availability and range of such training materials, for a diversity of audiences and addressing a diversity of RCR related topics, is growing too. This is of great help to all who are looking for materials to set up their RCR training, but how can one select what topics to include within the often-limited time available for training? In this paper we propose a step-by-step approach to set the agenda for RCR training, using the target audience as a narrative guide. The process consists of six steps: (1) mapping the needs and translating them into learning objectives; (2) prioritizing, selecting, and combining objectives and themes; (3) demarcation of the training; (4) completion of the program; (5) development or adaptation of training materials; and (6) pilot and evaluation. The development process of a training program for researchers in Dutch Universities of Applied Sciences is used as an example to illustrate the step-by-step process.
Introduction
Training in responsible conduct of research (RCR) is seen as a strong strategy to foster RCR and to work against research misconduct (De las Fuentes et al. 2005; Kligyte et al. 2008) and questionable/detrimental research practices (Geritts et al. 2019; Artino et al. 2019). In response to that, the availability and range of such training materials are growing. Path2Integrity (Path2Integrity 2022), VIRT2UE (VIRT2UE 2022), and INTEGRITY (H2020 INTEGRITY project 2022) are leading platforms that develop and share training materials on research integrity (RI) and RCR. These platforms focus on wide-ranging target groups from secondary school students to career researchers. Next to these large providers, there are smaller or even individual initiatives that share educational material, such as videos, knowledge clips, case studies, and workshops. These initiatives are collected on platforms like The Embassy of Good Science (The Embassy of Good Sciences 2022), onlineethics.org (Online Ethics Center 2022), or the European Network of Research Integrity Offices (ENRIO 2022).
Most work on RCR has been developed in academic settings. In 2018, for the first time in Dutch history, the Universities of Applied Sciences (UAS) signed the new version of the Dutch Code of Conduct on Research Integrity, thereby committing to institutional duties of care, including RCR training. So far, little has been done in UAS in the Netherlands to promote responsible research practices or to offer systematic training on this topic. This means that UAS researchers are often dependent on training programs offered by the research universities. However, there is reason to believe that researchers at UAS may have different training needs than those at academic centres. Their research is often executed in close collaboration with industry, and firms and managers often play an important role in the design and development of research. Moreover, many societal innovations are driven by decisions made by such managerial staff within the private sector, a domain where other ethical considerations may also come into play (Martin et al. 2019).
Although both academic and applied research share some common characteristics, there are some essential differences. Unlike academic research, the scope of UAS’s research is primarily on professional practice (innovation) and professional exercise and it is commonly in collaboration with industry, in particular with small and medium-sized enterprises (De Weert and Leijnse 2010). This type of research is deeply embedded within a particular social context and therefore requires a different subset of sensibilities when it comes to RI and ethics where social accountability plays a central role. Related to this, a strong tradition of “action research” within UAS that puts both knowledge production and driving social change with multiple stakeholders at its core (McNiff 2013). Altogether, research at UAS is an undertaking that comes with its own set of ethical considerations (Martinuzzi et al. 2018; Cohen et al. 2017; Brydon-Miller 2012). In this regard, the already existing RCR training programs have a strong academic focus (Pizzolato et al. 2020), which lacks the aforementioned hallmarks of applied research that is essential to UAS.
As RCR training should be tailored to the target group (Anderson et al. 2012; Labib et al. 2022), should address the needs of specific day-to-day research practices, and should be a bottom-up process (Zwart and Ter Meulen 2019), it is important to tailor training to UAS researchers and therefore involve them early on in the process of training development. How does one achieve that?
To address this, the TETRIAS project (Translating Researchers’ Experiences into Training on Research Integrity at universities of Applied Sciences (TETRIAS 2022a)) has been set up as an initiative to involve UAS researchers in the development of a tailor-made RCR training. A series of interviews with 12 UAS researchers gave insights into how they manage integrity issues in their daily research and what their current needs are on this front. This paper serves as an inspirational guide on how one can develop RCR training programs through a bottom-up approach, using the target audience of the training as a narrative guide. To aid future researchers in developing similar materials, we illustrate a step-by-step approach in developing a tailored RCR training program with TETRIAS as an example.
A six-step approach
The development of a new training program, or the redesign of an existing training program, usually starts with a reason and, where available, it usually builds on previous experience. In this six-step approach, however, building on previous experience is postponed, and the aim is to start the process with the needs of the future trainee and avoid any existing training that may dictate the content of the new training.
Central to this approach for the development of RCR training are two questions:
1.
What do we want to teach to whom?
2.
How can we best do that?
This first central question includes who is the target audience, what are their training needs, and what competencies would we like to train this audience to achieve the desired outcome? Together those answers give rise to the learning objectives of the training. The second central question should be answered from three perspectives: how can we best meet (i) the learning objectives, (ii) the wishes and needs of the trainees, and (iii) the skills and preferences of the trainers?
These central questions led to the following steps, and are to be reflected on while taking the six steps:
1.
Mapping the needs and translating them into learning objectives
2.
Prioritizing, selecting, and combining objectives and themes
3.
Demarcation of the training
4.
Completion of the program
5.
Development or adaptation of training materials
6.
Pilot and evaluation
Step 1: Mapping the needs and translating them into learning objectives
Developing or designing a training program starts with understanding the reason for this training, the current needs of the target group, and the experiences and limitations of the target group regarding RI. This is fundamental to answering the first central question: what do we want to teach to whom? Before the objectives of the training can be drawn up, more insight is needed.
First, one must gain insight into the needs regarding the training’s content. What exactly are the needs that this training is to address? Once you have this insight, you should reflect: is this a proper reflection of the situation or is there more to understand the underlying need that the training should address? Second, one must decide if it is feasible. There are often certain conditions at play that can restrict the training, such as how long it will take; in what language(s) it should be offered; whether it should be online or on location, or both; etc. Also, will the training be mandatory or optional? Next to such conditions, the target audience also influences the feasibility. Moreover, the target audience might set some conditions too. How easy can you reach them and how much time are they usually willing or able to spend on such training? Then, take a step back: who is or who are the target audience(s)? If there are multiple audiences, which of their needs are the same and what relevant differences should be considered? This might lead to the decision to offer different training options (e.g., per research domain) or levels (e.g., basic and advanced training).
Once clarity is reached on the issues above, we can start formulating potential learning objectives. Further on in Step 3 (demarcation of the training) it will be decided which of these potential learning objectives will be selected for the training program. The potential learning objectives rise from the perspective of the target group and the expertise of the training developers. These learning objectives will help steer the development process, and the more sharply they are formulated, the more guidance this will give you during the rest of the process.
In the TETRIAS project, we used Bloom’s taxonomy (Bloom et al. 1956; Anderson et al. 2001) to help us formulate the learning objectives. This taxonomy distinguishes six cognitive domains, and for each cognitive domain lists of verbs are available to formulate learning objectives that match that domain. For example, if you aim for the cognitive domain “applying”, the learning objective should rather be phrased as “determine which factors influence research climate” than “name factors that influence research climate” (“naming”, or “recalling” is listed in the domain “remembering”).
Finally, a list of themes or overarching topics can be compiled. For example, a learning objective like “The trainee can illustrate the scope of ‘research integrity’ with examples of breaches of research integrity” is rather broad and requires having a clear understanding of what this scope is and which topics within that broad scope are relevant to the target audience. It is important to not only map the topics that the interviewees address but also what they don’t address (the gaps).
To summarize, mapping the needs includes mapping of four aspects: (i) conditions/restrictions (time, language, optional, online, etc.), (ii) target audiences, (iii) objectives, and (iv) themes and requires insight in the training demand and target audience and expertise of what constitutes RCR.
Step 2: Prioritizing, selecting, and combining objectives and themes
The long list of wishes and needs must be prioritized. What can and cannot be trained? Which objectives are easy to achieve and which require more time and effort? What should the focus be?
One of the criteria for priority setting would be the skills and attitude deemed necessary for the RCR objectives. Not only should these skills and attitudes be higher priority, but also the knowledge and motivation that are required to work on these skills and attitudes. A second set of criteria can be found in the conditions and restrictions resulting from Step 1. Additionally, which objectives and themes can be combined? Making a mind map and clustering can be a useful tool for this step; also highlight the topics that the interviewees did not address but that are deemed important by experts. What is not discussed might determine exactly where more awareness is needed.
Next, a visualization of the alignment of the draft objectives and themes can give insight in the completeness of the draft selection made. A cross-table of objectives (vertically) and themes (horizontally) can provide insight in the themes that are well covered and the themes that might require more attention in this draft selection. See for example Table 1.
Table 1.
| RI knowledge | Attitude and skills | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Examples | Scope of RI and demarcation of that scope | Relevance of RI | Risk- and protective factors | Research climate | Rules and regulations | Prevention | Unconscious incompetence | Ethical decision-making | Moral compass | Reflection | Open conversation/dialogue | RI is a joint process | RI stewardship |
| Research waste | √ | √ | √ | √ | √ | ||||||||
| Reporting | √ | √ | √ | (√) | √ | ||||||||
| Authorship and publishing | √ | √ | √ | (√) | √ | ||||||||
| Competition, hierarchy, and favouritism | √ | √ | √ | √ | |||||||||
| Conflict of interests | √ | √ | √ | √ | √ | √ | √ | √ | |||||
| Intellectual property | √ | √ | (√) | ||||||||||
| Dealing with dilemmas and potential breaches of RI | (√) | (√) | √ | (√) | (√) | √ | √ | √ | √ | (√) | (√) | ||
| “Swampy lowlands of practice” | √ | √ | √ | √ | √ | ||||||||
| Not explicitly mentioned, but relevant | |||||||||||||
| Supervision and role-modelling | √ | √ | √ | √ | √ | √ | √ | √ | |||||
| Work–life balance and mental health | √ | √ | √ | (√) | √ | √ | |||||||
| Open science | √ | (√) | √ | √ | √ | √ | |||||||
| Limited | |||||||||||||
| Data management and privacy | √ | √ | (√) | (√) | (√) | ||||||||
| Methodology and statistics | √ | √ | √ | (√) | (√) | √ | (√) | ||||||
| Strict ethical issues | √ | √ | √ | (√) | |||||||||
Note: Checkmarks between brackets indicate less direct links, but links can be made. RI, research integrity.
Step 3: Demarcation of the training
In the this step final decisions on the objectives and themes are made. Again Bloom’s taxonomy can be of help to select the final objectives. For example, basic training is more likely to address the lower and middle cognitive domains than in-depth training, which might trigger the higher cognitive domains; training for undergraduate students would address lower cognitive domains than training for experienced researchers, of whom higher cognitive levels should be expected. Also, when training for a specific subgroup of a target group is developed, the topics addressed in this training could also be specifically selected for that group. Additionally, more advanced themes could be programmed for later training sessions to let the program’s level develop along with the learners. Furthermore, the conditions and feasibility of the training play a decisive role in this step.
Step 4: Completion of the program
Now that the scope of the training is set, the second central question “how can we best do that?” is addressed. For each learning objective suitable training methods are listed. Next, cases and examples matching the selected themes are collected. The abovementioned collections of training materials are useful sources for existing materials and to find inspiration. It might be useful to first cluster the learning objectives so that objectives that are likely to be met by similar training methods are clustered together. To maximize the transfer of theoretical skills to their everyday research practices, it is important that the training methods employed in the training fit the learning goals, which is a key aspect of constructive alignment (Biggs and Tang 2011).
Subsequently, a selection is made out of the list of training methods in such way that each objective is addressed by at least one training method in the selection. From these selected training methods a program is constructed, starting with the more cognitive elements and building up to more intensive methods. Suitable cases and examples can then be chosen for each training method where appropriate.
Step 5: Development or adaptation of training materials
The program is worked out in detail, including a syllabus for trainees and a handbook for trainers and for back-office support. For each training method in the program, it is described what the aims are, what both the trainees and trainers should do, and what the potential outcomes of exercises could be. Such outcomes are, for example, which courses of action and argumentation one could have in particular cases or potential solutions to a problem. Again, for an optimal training effect the outcomes of the training methods should be aligned with the goals of the training and with everyday practice of the training participants (Biggs and Tang 2011). In preparing the training materials keep the method of delivery in mind; are the training sessions online or on-site, for example?
Step 6: Pilot and evaluation
Prior to conducting the training, a pilot is useful to optimize and finalize the training. The pilot can be done with a small group of participants, whereby feedback on different aspects of the training will be collected (for example, on the preparatory tasks, the instruction manual, the quality of the trainers, or the coherence of the program). On the other hand, trainers will also collect information about the experiences with the training materials; does everything work out as planned? Please keep in mind that the participants in a voluntary pilot are likely to be more positive towards the topic or are already aware of the relevance of it. If the training is to be mandatory in the future, responses from the participants may differ.
To ensure the predetermined learning objectives are achieved by the participants, the participants have to show their learning outcome. By building a portfolio, the participants can show their professional and personal development. In a portfolio, for example, reflections can be written about theoretical teaching material, or a personal experience, or group discussion during a class meeting. Feedback from the trainer and fellow participants can also be added. Portfolio assessments are seen as an appropriate way to assess the participant’s attitudes; it stimulates reflection, critical thinking, and gives participants the accountability for their own learning process (Davis and Ponnamperuma 2005). As every participant has their own background and research experiences, they can tailor their portfolio to their specific needs—this will also stimulate the intrinsic motivation.
In addition to the substantive evaluation of the participant’s progress, a process evaluation at the end of every training session helps improve the quality of the training.
Six-step approach applied: the TETRIAS project as an example
In the TETRIAS project we took ample time to prepare for the development of the training program. As part of Step 1, an interview study was conducted (approval was obtained from the Research Ethics Committee of the HAN University of Applied Sciences, file no. ECO 185.04/20). We asked the advisory board members of the project to recruit researchers from their UAS for an interview. We asked them to recruit one junior and one senior researcher from specific research domains, to get a diverse group of interviewees. This resulted in 12 interviews with participants form seven different UAS, ranging from doctoral student to professor from six different research domains (e.g., economics, health, and education).
The interview guide consisted of six topics:
1.
Good research and good researcher (e.g.,: “What characterizes a good researcher?”)
2.
Cracks in science (e.g.,: “Do you ever see situations where you feel like things are not going quite right?”)
3.
Netherlands Code of Conduct for Research Integrity (e.g.,: “How recognizable is what is in the code of conduct in your research practice?”)
4.
Training in RI (e.g.,: “In what area do you think training can be of most benefit?”)
5.
Reflection (e.g.,: “Are there things you do yourself to be, become or remain a good researcher?”)
6.
Closing remarks (e.g.,: “What advice would you like to give us for the development of the training?”)
The complete interview guide is available via the project’s preregistration on the Open Science Framework (TETRIAS 2022b).
To ensure the quality of the research the interviews were conducted by two researchers; one took on the role of interviewer and the other researcher was the observer (researcher triangulation). After transcription of the audio files a member-check was performed. Finally, the data-analysis was performed independently by two researchers. Discrepancies were discussed until consensus was reached.
The transcripts were analyzed deductively using direct content analysis (Hsieh and Shannon 2005). For this purpose, the items from the interview guide served as prearranged codes. Coding was done by two researchers working independently, who first applied the prearranged codes and then further refined and completed them. Labelling was discussed until consensus was reached. One of the analysing researchers was either the interviewer or observer, while the other researcher was not present during the interview at hand. This resulted in an extensive code tree of more than 300 codes across 21 code groups (interim results code tree in Dutch available from the corresponding author on request). Results from this study are presented below to illustrate how the six-step approach is applied to the TETRIAS project.
Step 1: Mapping the needs and translating them into learning objectives
Conditions
The interviewees were not unanimous with regards to the conditions of the training. For example, some mentioned the training should be mandatory so all researchers get trained at some point, while others emphasized it should not be mandatory to take the training. Overall, the interviewees agreed the training should have a practical approach (quote 1), with practical examples and solutions. Some interviewees said the training should not be a one-time event only, but should have a structural character (quotes 2 and 3).
Quote 1: “It would be good, I think, if such training would point out this code of conduct and explain that there was really an attempt to make it as comprehensive as possible (…) for the entire academic community, and to invite them to reflect on the question ‘well, when it comes to independency, what does independency mean when you are an artistic action-researcher, how would you translate that?’ And that might be different from what the code of conduct suggests. But in that questioning way, and that reflective way, and to really link it to the type of research you’re doing and the standards that matter to your community, and your own moral compass, that I think would be good, to do it that way.”—Senior researcher, arts & culture
Quote 2: “As far as ethics is concerned, the most important thing is to ensure that it does not become incidental, something you have to do once in a while, but that it takes on a structural character.”—Senior researcher, education
Quote 3: “I actually think with research, or while conducting research, you never stop learning.”—Senior researcher, arts & culture
Target audience
Some different needs were expressed when discussing who should be the target audience of the training. Some respondents tend to assume it would be targeted at junior researchers, but when asked further questions, they would prefer more senior researchers to engage as well if that would be realistic. Also, several interviewees said that training in research teams would be of added value compared to individual training (quotes 4 and 5), although the success is dependent of the group dynamics (quote 6).
Quote 4: “A group learns much better than an individual, especially about complex issues.” —Senior researcher, economics
Quote 5: “Preferably you just want to have an entire team or from one study program in your course, so that there is a dialogue between them and that a vision arises.”—Senior researcher, education
Quote 6: “When mixing junior and senior researchers, it is very much dependent on how such a senior researcher positions him/herself, because if he/she starts to argue ‘yes, I know all about that and I will tell you how to do it’, then of course it doesn’t work. But I think you can learn from each other, so the senior researcher specifically from the questions asked by a junior researcher, and the junior researcher from the experiences of the senior researcher.”—Junior researcher, beta-technology
Learning objectives
Most interviewees indicated that general knowledge about RI and its relevance should be covered by the training (quote 7). Some also thought that including some theoretical background on ethical decision-making would be of added value. Others only mentioned decision-making when talking about training skills. Other relevant skills the interviewees would like to practice in RCR training are critical thinking about RCR and values/virtues relevant to RCR, dealing with different interests, and how to start and conduct team discussions or team reflection on RI-related issues. The following topics surfaced regarding attitude: awareness, following your moral compass or values, internalization of codes of conduct, and that RCR is something you don’t do individually but all together (quote 8) and that having conversations about RCR is key (quotes 9 and 10).
Quote 7: “I think it’s really important to figure out what the relevance is. Why integrity is so important. Why these are not rules that are imposed on the research, but that it helps you, and that it is also a form of respect towards the people you work with.”—Junior researcher, arts & culture
Quote 8: “I think it’d be good and that more courage [to ring the alarm bells] surfaces when you have a group of people around to discuss these things with, people who also struggle with that and then you support each other. And I don’t see this happening that much. It’s often like ‘what should I do as researcher, what needs to be done, I need to submit that proposal’. And in principle a virtue is an individual quality. While I would say: working together and doing it with the goal to enhance your virtues, I think that would actually be really, really important.”—Senior researcher, education
Quote 9: “There are always situations in which you have doubts, but then it is a matter of consulting with a colleague and then weighing the pros and cons, and then you come to a decision.”—Junior researcher, beta-technology
Quote 10: “When you receive criticism from others, this automatically reduces your risks of spinning out, and maybe that’s the most important, that you have conversations about that, and yeah, may ask some questions.”—Senior researcher, beta-technology
The objectives expressed by the interviewees were labelled and clustered, resulting in the following list of elements to be addressed by the learning objectives:
Knowledge:
•
Scope of RI, and demarcation of that scope
•
Relevance of RI
•
Risk factors and protective factors
•
Research climate
•
Rules and regulations (e.g., regulations on research with human subjects, standard operation procedures, complaints procedures and whistle-blower protection)
•
Prevention
Attitude and skills:
•
Unconscious incompetence
•
Ethical decision making
•
Moral compass
•
Reflection
•
Open conversation/dialogue
•
RI is a joint process
•
RI stewardship
Themes
The interviewees named various topics as relevant parts of RI. The themes that they specifically mentioned when discussing the content of RCR training or that were apparent in the examples they told us, are:
•
General knowledge (scope and relevance of RI)
•
Research waste
•
Competition, hierarchy, and favouritism
•
Data management and privacy
•
Authorship and publishing
•
Reporting
•
Methodology and statistical significance
•
Intellectual property and conflicts of interest
•
Dealing with dilemmas and potential breaches of RI
•
The often challenging and rapidly changing context of practice-based research, the so-called “swampy lowlands of practice”, in which UAS researchers must do their work
These themes are partially in line with most other available educational resources. The first theme on our list, addressing the scope and relevance of RI, is moving away from the focus on research misconduct, which is the primary theme among the available resources (Pizzolato et al. 2020).
What was striking was that the focus was often on data management, privacy and regulatory issues, and that a need for tailored RCR training was underlined by the notion that research done by UAS has an extra dimension that makes it more complicated (quotes 11 and 12). Research is performed in collaboration with external partners, needs to have societal relevance, and seldom is the main priority. This contributes to the feeling expressed by most interviewees that there is still a lack of a research culture in most UAS (quotes 13 and 14). Moreover, working on research culture can contribute to the attitude that RCR is not an individual endeavour but something you do together in your team (quotes 15 and 16).
Quote 11: “I think that is also the essence of practice-based research, that practice can make use of it. And that mainly influences the scope of your research, what you are going to tackle, but sometimes also with which data you are going to work, which methods are you going to apply. That will of course trickle down.”—Senior researcher, beta-technology
Quote 12: “Lectors [full professors at UAS] who have not continued their professorship, but who have dropped out prematurely. Then the cause of this often lies in the absence of that connection with practice. And I think that this connection with practice is higher on the list of priorities than a high-quality publication.”—Junior researcher, food and agriculture
Quote 13: “What do you call that, the psychologists and sociologists, to internalize, I believe right? So that you really make these kinds of norms your own and consider that evident and apply it to everything that you do, or at least as far as possible, eh, for us that is not yet the case. And I think it should.”—Junior researcher, economics
Quote 14: “The culture, that of course, is another. At research universities the culture is fully organized around research. Education is organized around research, and so is the organization as a whole. And within UASs you can see that the organization is fully organized around education. Well, of course we are slowly changing in that regard but that just requires several steps which are related to research and to embed that research culture more firmly in the organization.”—Senior researchers, beta-technology
Quote 15: “I think maybe that you not only learn how you as an individual can live up to the codes of conduct, and do research with integrity, but that you also will learn about how to take care of a healthy system within the research group, and how to maybe guide that. That would be a bit more innovative in my opinion, than what you can do by yourself.”—Senior researcher, health and care
Quote 16: “But sometimes that goes against doing good research, and then you’d have to dare to ring the alarm bells. That is also a part of my research, that you think ‘yes but’, we shouldn’t think of that too individualistically. That courage really also comes from those fellow researchers too. You’d have to organize that well, to give rise to more integrity.”—Senior researcher, education
Step 2: Prioritizing, selecting, and combining objectives and themes
The prioritization is not only guided by the input from the interviews, but also by expert opinions and the training developers’ expertise and experience. An advisory group of the TETRIAS project, consisting of representatives of eight large UAS and the Association of UAS, gave feedback on the results of Step 1.
In addition to the themes listed above, we decided to add three themes: supervision and role-modelling, work–life balance and mental health, and open science. These were also mentioned by the interviewees, but not directly when talking about training.
Data management and privacy, methodology and statistics, and strict ethical issues (which are not considered RI issues, e.g., regarding the use of laboratory animals or risks to study participants) were themes frequently mentioned by interviewees; however, these will not get a prominent role in the training. Training materials for these topics are already available. Addressing these topics in sufficient detail will take up scarce time available for the training we are developing. The relevant knowledge may also differ largely per institution, e.g., on data management facilities. So rather than including these themes, the training aims to inspire participants to seek further training on those topics. During the discussion of cases, the importance of such topics will be felt by the trainees and that is when the trainers can refer to other trainings and, where applicable, relevant facilities in the institution. The skills and empowerment that this training aims to stimulate will benefit the researchers in seeking further training.
Table 1 shows the possible link between the expressed objectives by interviewees and the themes. Themes that are similar to topics already listed in the objectives have been removed. This overview served the development of the training by giving insight in themes or objectives that might be lacking and to which extra attention might be sought.
Step 3: Demarcation of the training
The input for the learning objectives of the training was rather broad, addressing many relevant potential aims of such training. Covering all these objectives in one training unit would result in an extensive course, which would exceed the conditions and restrictions that need to be considered. However, since the interviewees also listed a variety of target audiences, and some even explicitly mentioned the idea of having different training for different audiences and objectives, this allowed us to explore the scenario of developing a set of different trainings. Two trainings were felt to be most relevant on a short notice, whereas some other trainings might be further developed in later years. Within the scope of the TETRIAS training the priority was set to develop a basic RCR training for individual UAS researchers and, if time allows, to also develop an advanced training for research teams.
The learning objectives selected for these two trainings, and the cognitive domains these objectives speak to, are shown in Tables 2 and 3. To ensure that these learning objectives cover all the objectives and themes resulting from the previous steps, the checkmarks in Table 1 were replaced with the numbers of the learning objectives (available on request from the corresponding author). This showed that there were no gaps left.
Table 2.
| Learning objective—after this training the participant can: | Cognitive domain | |
|---|---|---|
| 1 | Understanding | |
| A | Exemplify the scope of the theme “research integrity” (RI) with examples of breaches of RI, and | Understanding |
| B | Express the relevance of RI for research, researchers and society | Understanding |
| 2 | Applying | |
| A | Recognize and define risk factors and protective factors for RI | Remembering |
| B | in casuistry, and | Applying |
| C | in one’s own research | Analysing |
| 3 | Applying | |
| A | Display the understanding of RI (objective 1) in terms of values of good research and a moral compass, and | Applying |
| B | articulate these in conversation | Applying |
| 4 | Analysing | |
| A | List responsibilities for RI of diverse stakeholders in research, and | Analysing |
| B | illustrate how these stakeholders can foster RI | Analysing |
| 5 | Evaluating | |
| A | Reflect on one’s own skills and responsibilities for RI, | Evaluating |
| B | compose matching actions, and | Creating |
| C | integrate in those actions relevant facilities, such as protocols and advisory bodies, and | Analysing |
| D | engage in open conversations about responsibilities and actions | Evaluating |
Table 3.
| Learning objectives: after this training the team and team members can: | Cognitive domain | |
|---|---|---|
| 1 | Analysing | |
| A | Identify research integrity (RI) issues in own work, and | Analysing |
| B | enable discussion of such issues | Analysing |
| 2 | Analysing | |
| A | Give interpretation to RI stewardship, by | Evaluating |
| B | identifying personal values and | Analysing |
| C | values that are important to the team | Analysing |
| 3 | Evaluating | |
| A | Engage in responsible decision making by | Evaluating |
| B | engaging in a constructive dialogue on RI issues, and | Evaluating |
| C | interpreting and weighing values that the team deems important | Evaluating |
| 4 | Creating | |
| A | Compose joint actions that | Creating |
| B | raise awareness of RI issues, and | Applying |
| C | enable discussion of the theme, and | Analysing |
| D | eevelop a professional atmosphere that fosters RI | Creating |
Step 4: Completion of the program
First, all learning objectives were translated into practical descriptions of the knowledge the trainees should have or the behaviour they engage in after completing the training. Among these descriptions a couple of clusters could be identified. The clusters made it easier to compile the lists of suitable training methods without doubling the work for similar objectives.
Second, from the lists of suitable training methods a selection was made, covering each cluster of objectives, to assemble the training programs (Tables 4 and 5). The learning objectives and programs were presented to the project’s advisory group for feedback.
Table 4.
| Meeting no. | Preparation by participants (approx. 1 h for each meeting) | Activities during the meeting (3 h per meeting) |
|---|---|---|
| 1 | 1.1 Mind map 1.2 Online module 1.3 Code of Conduct 1.4 Personal learning objectives | Introductions 1.5 Mind map 2.0 1.6 Knowledge quiz 1.7 Dilemma game |
| 2 | 2.1 Exercise “what is a good researcher?” 2.2 Exercise “responsibilities” | 2.3 Discussion of 2.1 using Mentimeter 2.4 Values of good research in fictional cases 2.5 Subgroup discussions of 2.2 2.6 Plenary recap of 2.5 |
| 3 | 3.1 Online module on research under current research conditions 3.2 Concept version self-reflection (Note: self-reflection is finalized after meeting 3) | 3.3 Cases on research climate 3.4 Mind map 3.0 3.5 Reflection in subgroups 3.6 Harvest of take-home messages Closing and evaluation |
Table 5.
| Meeting no. | Preparation by participants (approx. 1 h for each meeting) | Activities during the meeting (2 h per meeting*). |
|---|---|---|
| 1 | 1.1 Summary of online module “Introduction to research integrity” (RI) 1.2 Team discusses their learning goals | 1.3 Inventory of learning expectations, following Disney method (Dilts 1994) 1.4 Debate–dialogue exercise 1.5 Mind map of RI themes, following TOVEREN dialogical method (Kessels and Boers 2022) |
| 2 | 2.1 Personal reflection on “what are good research, a good researcher, and a good research team?” 2.2 Personal case description | 2.3 Compiling profiles of good researchers and a good research team 2.4 Demo of CURA dialogical method (CURA 2022) |
| 3 | 3.1 CURA in small groups | 3.2 Reflection on CURA exercise 3.3 Advanced dialogical method (Moral Case Deliberation, or an adapted combination of Virtues & Norms and The Middle Position) |
| 4 | 4.1 Mind map of research culture | 4.2 Mind map discussion and prioritization (agenda setting) 4.3 Brainstorm sessions 4.4 Joint plan of action |
*
Each meeting starts with a check-in question and ends with reflection and harvest of take-home messages.
Step 5: Development or adaptation of training materials and Step 6: Pilot and evaluation
For both the basic training for individual researchers and the advanced training for research teams, instruction manuals for both trainees and trainers have been developed (in Dutch, available upon request from the authors).
The basic training for individual researchers was tested in a pilot with nine senior researchers from six different UASs. Due to the COVID-19 pandemic the training sessions were organized online. Based on their feedback on the content and process, a number of changes were made to the training design. An example is that the assignments took longer than initially planned. In response, some assignments are excluded or minimized or the indicated preparation time extended for exercises that the participants indicated were relevant and worth the effort.
At the end of the pilot all participants submitted the portfolios, which included several assignments and peer feedback. The trainers read those documents carefully and when satisfied, the participant received a certificate.
Discussion
To develop tailor-made training for researchers in UASs, we implemented a step-by-step strategy. The program is based on the results of a series of 12 interviews with the target audience and member-checked by the advisory group of the TETRIAS project. Researchers from seven different UASs were involved. We consider this bottom-up approach for the training development a strength of the project and expect that this will contribute to the uptake of the training.
The interviewees represent six different research domains. In UAS there are seven different domains, this means one was not included in the interviews (social studies). Nevertheless, we feel we got a good view of the target audience and their experiences. Across the research domains we found similarities in the difficulties of “doing research in the swampy lowlands of practice” and the lack of a research culture. Also, the differences among the interviewees confirmed the broadness of the needs that the training should aim to address. Adding more interviews with researchers from a different research domain would probably not have altered those conclusions, as these conclusions are on a more fundamental level that transcend the domains. We therefore consider these 12 interviews sufficient for the purpose of the project.
What is different across research domains, is the research itself. For the training to be appealing to all participants, the cases used ought to be either about general issues that occur across all domains or should be easily adaptable or translated to the participants research vocabulary. Paying attention to this is not specifically part of the presented six-step approach but could very well be made an explicit part of the train-the-trainer sessions. Each trainer can reflect on the suitability of the cases for their own research domain and think of how to adapt it if necessary. Regarding the TETRIAS training we aim to collect the proposed alterations with the aim to compile an inspirational collection for other trainers to use too.
The question remains, can RI be taught? As expressed by our interviewees and in the literature, it doesn’t take individuals but a research culture that enables and stimulates individuals to be responsible researchers (Zwart and Ter Meulen 2019; Haven 2021; Aubert Bonn and Pinxten 2021). Training can contribute to fostering such a research culture and to give researchers the knowledge, tools, and empowerment to both contribute to such a research culture and to act with integrity. Training is not the single solution to RI, but it can greatly contribute, especially when a growing group of researchers engage in training and bring those lessons to their research practice.
Recommendations
Based on our experiences with this six-step development process, we make some recommendations to others who are planning to develop similar trainings.
First, we recommend taking ample time for this process, especially Step 1 (inquiring and analysing the experiences, needs, and wishes of the target audience), as it is a time-consuming process. The interviews were very useful to our project and therefore also highly recommended as part of Step 1. Go back to the data frequently. Use an iterative approach to draw up the results of Step 1. For each part of Step 1 (the conditions, target audiences, objectives, and themes), consult the transcripts again.
Second, during the training development process you need to make many decisions, some more evident than others. Therefore, installing an advisory group that can give feedback and offer advise at several time points during the process is helpful. The advisory group also serves as a member-check of the interpretations made and ways in which we translated those into the training program.
Next to an advisory group, advice can also be sought from an expert in didactics. In the TETRIAS project we found this helpful because of the reflective questions asked by the expert and the practical suggestions and other inspiration for teaching methods.
Besides including an expert on didactics, be an expert on training on the topic at hand or include such an expert in the training developing team. Knowledge from other related training programs and experience with a wide variety of training methods used in training on the topic is highly recommended.
Finally, plan and think about the training of trainers and the implementation and financing of the training ahead of time. When designing the training program, certain implementation aspects should be considered. Think, for example, about the potential practical bottlenecks in submitting preparatory assignments by participants.
Conclusion
The six-step approach to develop tailor-made training using the target audience as a narrative guide is a useful and recommendable approach. The success of this approach for the TETRIAS project has yet to be seen as the pilot and implementation have not started at time of writing this paper. Nevertheless, we have experienced this approach as a useful, structured method for this project.
Acknowledgements
We like to thank the members of the advisory group of the TETRIAS project, all interviewees, and the participants in the pilot training for their contribution and words of wisdom. We learnt a great deal from you!
References
Anderson LW, Krathwohl D, Airasian P, Cruikshank KA, Mayer RE, Pintrich P, et al. 2001. A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s taxonomy of educational objectives. Addison-Wesley Longman, Boston, Massachusetts.
Anderson EE, Solomon S, Heitman E, DuBois JM, Fisher CB, Kost RG, et al. 2012. Research ethics education for community-engaged research: A review and research agenda. Journal of Empirical Research on Human Research Ethics, 7(2): 3–19.
Artino AR, Driessen EW, and Maggio LA. 2019. Ethical Shades of Gray: International Frequency of Scientific Misconduct and Questionable Research Practices in Health Professions Education. Academic Medicine: journal of the Association of American Medical Colleges, 94(1): 76–84.
Aubert Bonn N, and Pinxten W. 2021. Rethinking success, integrity, and culture in research (part 2) — a multi-actor qualitative study on problems of science. Research Integrity and Peer Review, 6(3).
Biggs J, and Tang C. 2011. Teaching for Quality Learning at University. 4th ed. Open University Press, Berkshire.
Bloom BS, Engelhart MD, Furst EJ, Hill WH, and Krathwohl DR. 1956. Taxonomy of educational objectives: the classification of educational goals. Handbook 1: Cognitive Domain. Longman, Green and Co, New York, NY.
Brydon-Miller M. 2012. Addressing the ethical challenges of community-based research. Teaching Ethics, 12(2).
Cohen L, Manion L, and Morrison K. 2017. Research Methods in Education. Routledge
CURA: Concentreren, Uitstellen, Reflecteren, Actie ondernemen. 2022. [to concentrate, postpone, reflect and take action], [online]: Available from venvn.nl/thema-s/ethiek/cura/.
Davis MH, and Ponnamperuma GG. 2005. Portfolio Assessment. Journal of Veterinary Medical Education, 32(3): 279–84.
De las Fuentes C, Willmuth ME, and Yarrow C. 2005. Competency Training in Ethics Education and Practice. Professional Psychology: Research and Practice, 36(4): 362–366.
De Weert E, and Leijnse F. 2010. Chapter 11 Practice-Oriented Research: The Extended Function of Dutch Universities of Applied Sciences. In The Research Mission of Higher Education Institutions outside the University Sector Edited by S. Kyvik, B. Lepori, (Red.) Springer. pp. 199–217.
Dilts R. 1994. Strategies of Genius, In Cupertino. vol. 1. Metapublications, California.
ENRIO: European Network of Research Integrity offices. 2022. [online]: Available from enrio.eu/.
Gerrits RG, Jansen T, Mulyanto J, Van den Berg MJ, Klazinga NS, and Kringos DS. 2019. Occurrence and nature of questionable research practices in the reporting of messages and conclusions in international scientific Health Services Research publications: A structured assessment of publications authored by researchers in the Netherlands. BMJ Open, 9(5).
Haven TL. 2021. Towards a responsible research climate: Findings from academic researchers in Amsterdam, PhD, Vrije Universiteit, Amsterdam.
Hsieh H-F, and Shannon SE. 2005. Three approaches to qualitative content analysis. Qualitative Health Research, 15(9): 1277–1288.
H2020 INTEGRITY project. 2022. [online]: Available from h2020integrity.eu/.
Kessels J, and Boers, E. 2002. Vrije ruimte filosoferen in organisaties: Klassieke scholing voor de hedendaagse praktijk [Free space philosophizing in organizations: Classic education for contemporary practice]. Boom, Meppel.
Kligyte V, Marcy RT, Waples EP, Sevier ST, Godfrey ES, Mumford MD, et al. 2008. Application of a sensemaking approach to ethics training in the physical sciences and engineering. Science and Engineering Ethics, 14(2): 251–78.
Labib K, Evans N, Roje R, Kavouras P, Reyes Elizondo A, Kaltenbrunner W, et al. 2022. Education and training policies for research integrity: Insights from a focus group study, Science and Public Policy, 49(2): 246–266.
Martin K, Shilton K, and Smith J. 2019. Business and the ethical implications of technology: Introduction to the symposium. Journal of Business Ethics, 160: 307–317.
Martinuzzi A, Blok V, Brem A, Stahl B, and Schönherr N. 2018. Responsible Research and Innovation in industry-challenges, insights and perspectives. Sustainability (Switzerland), 10(3): 1–9.
McNiff J. 2013. Action research: Principles and practice. Routledge.
Online Ethics Center. 2022. [online]: Available from onlineethics.org/resources
Path2Integrity. 2022. Rotatory role-playing and role-models to enhance the research integrity culture. [online]: Available from path2integrity.eu/
Pizzolato D, Abdi S, and Dierickx K. 2020. Collecting and characterizing existing and freely accessible research integrity educational resources. Accountability in Research, 27(4).
TETRIAS: Translating researchers’ Experiences into Training on Research Integrity at universities of Applied Sciences. 2022a. [online]: Available from embassy.science/wiki/Theme:9de016c0-cdce-4438-bbb2-92698a8b3293
TETRIAS. 2022b. TERIAS Interview guide. [online]: Available from osf.io/4yfvt/
The Embassy of Good Science. 2022. [online]: Available from embassy.science/wiki-wiki/index.php/Special:BrowseData/Resource?_search_Resource_Type%5B0%5D=Education
VIRT2UE. 2022. Virtue-based ethics and Integrity of Research: Train-the-Trainer program for Upholding the principles and practices of the European Code of Conduct for Research Integrity. [online]: Available from virt2ueconsortium.eu
Zwart H, and Ter Meulen R. 2019. Addressing research integrity challenges: from penalising individual perpetrators to fostering research ecosystem quality care. Life Sciences, Society and Policy, 15(5).
Information & Authors
Information
Published In
FACETS
Volume 7 • Number 1 • January 2022
Pages: 1258 - 1275
Editor: Julia Prieß-Buchheit
History
Received: 30 April 2021
Accepted: 9 July 2022
Version of record online: 22 September 2022
Notes
This paper is part of a collection titled “Horizon 2020 Path2Integrity”.
Copyright
© 2022 Blom et al. This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Data Availability Statement
This manuscript reports on data that were collected and analyzed for a different purpose than this manuscript describes. The data are therefore not publicly available with this manuscript, but are available from the corresponding author on reasonable request, and will be included in a different manuscript by the authors.
Key Words
Sections
Subjects
Plain Language Summary
There is a growing interest in training on responsible conduct of research (RCR). The availability and range of such training materials is growing too. But how to select what to include in a training program?
We propose a step-by-step approach to set the agenda for RCR training, making use of the experiences of the researchers for whom the training is being developed (the target audience). The approach consists of six steps: 1) mapping the needs of the target audience and translating these into learning objectives for training; 2) prioritizing, selecting and combining learning objectives and RCR themes; 3) demarcation of the training based on the levels to address; 4) final completion of the training program; 5) development or adaptation of training materials; 6) pilot and evaluation. As an illustration of this six-step approach, we show the development process of a training program for researchers in Dutch Universities of Applied Sciences in this manuscript.
Plain Language Summary
Setting the training agenda for responsible conduct of research
Authors
Author Contributions
All conceived and designed the study.
All performed the experiments/collected the data.
All analyzed and interpreted the data.
FB and RS contributed resources.
All drafted or revised the manuscript.
Competing Interests
The authors declare there are no competing interests.
Funding Information
The TETRIAS project was funded by The Netherlands Organisation for Health Research and Development (ZonMw).
Metrics & Citations
Metrics
Other Metrics
Citations
Cite As
Fenneke Blom, Susan Maria Johanna Berentsen, and Rob van der Sande. 2022. How to set the agenda for training in responsible conduct of research using the target audience as a narrative guide. FACETS.
7: 1258-1275.
https://doi.org/10.1139/facets-2021-0048
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
There are no citations for this item