Introduction
Impact assessment (IA) is a widely adopted environmental governance mechanism that engages a variety of diverging perspectives and knowledge to identify potential positive and negative effects of a proposed development project (
Noble and Press 2011;
UNEP 2018). IA processes are generally organized through a designated primary assessment organization, which can be a centralized government agency, a sector-based agency, or a dedicated assessment board (
UNEP 2018). More decentralized models of IA utilize assessment boards as a boundary-spanning organization tasked with negotiating a variety of agendas into specific recommendations for development projects (
Partidario and Sheate 2013;
Meuleman 2015). IA boards are responsible for gathering, interpreting, and considering evidence from a variety of knowledge sources to formulate recommendations prior to regulatory approval, making IA a distinctly knowledge-based activity (
Noble and Press 2011;
Bond et al. 2018a;
UNEP 2018). However, the ability of these governance bodies and their employees to undertake the required knowledge gathering and management tasks—research capacity—heavily influences the effectiveness and legitimacy of the overall process (
Marsh and Smith 2000;
Pope et al. 2013;
Howlett and Ramesh 2015;
Bond et al. 2018b).
Research capacity is understood to be the ability of an actor, organization, or network to engage, produce, maintain, and use knowledge through individual and collective development (
Cashmore 2004;
Kolhoff et al. 2009;
Nykvist and Nilsson 2009;
Partidario and Sheate 2013;
Wright 2014;
Lonsdale et al. 2017). Research capacity therefore includes the ability to interact with knowledge beyond conventional academic or scientific research activities, which focus on knowledge production, to capture all interactions with knowledge and information. The concept has been discussed in professional fields that rely heavily on the transmission of knowledge between organizations and individual practitioners, such as public health and international development (e.g.,
Cooke 2005;
Mugabo et al. 2015;
Maag et al. 2018), yet in environmental governance the concept has rarely been engaged directly (e.g.,
van der Molen 2018;
Rahman et al. 2019;
Gustafsson et al. 2020). In this paper, we consider IA to be a knowledge-intensive governance mechanism that relies on the ability of the assessment body and their assessors to not only produce new information and knowledge, but also to maintain, use, and interpret the knowledge and information provided by other participants. Previous IA research acknowledges the need to address different perspectives, but tends to focus on the coordination of technical information rather than value-driven knowledge (e.g.,
Bond et al. 2018a;
Darling et al. 2018).
In northern Canada, equal consideration of diverse worldviews in development activities is a public policy priority, with particular focus on IA as a mechanism to ensure inclusion in decision-making (
UFA 1993;
Sabin 2016;
Southcott et al. 2018). There are three territorial review boards in Canada, all of which have generated interest among academics and practitioners since their respective establishments through regional land claims processes: the Mackenzie Valley Environmental Impact Review Board (Northwest Territories), the Nunavut Impact Review Board (Nunavut), and the Yukon Environmental and Socio-Economic Assessment Board (YESAB; Yukon) (e.g.,
Fitzpatrick et al. 2008;
Gondor 2016;
Peletz et al. 2020). These northern IA bodies are relatively well-funded through their foundations in land claims agreements, with a high degree of the legislative legitimacy that remains a challenge elsewhere, even in Canada (
Bond et al. 2018a). Particular attention has been drawn to their consideration of Indigenous rights and title and Indigenous and Traditional Knowledge (White 2009). At the forefront of criticisms are challenges associated with the related process of territorial devolution of power and limitations on the flexibility of IA in considering nontechnical and nonscientific knowledge (
Christensen and Grant 2007;
Gondor 2016). While all three boards share foundations in regionally specific land claims processes, each has experienced an evolution in their roles as broader devolution negotiations and implementation unfold, including their approach to pluralism in IA processes.
Pluralism—the active engagement of the multitude of worldviews, approaches, and solutions to societal problems—is a major consideration in IA, where a variety of theoretical, methodological, and representational perspectives are included in assessment processes (e.g.,
Cape et al. 2018). Yet, little empirical work has been done to evaluate the extent to which assessment boards have the capacity necessary to realize desired outcomes, namely the active engagement of a plurality of worldviews to the benefit of sustainable development (e.g.,
Peletz et al. 2020). In this paper, we present the case of the YESAB as a specialized organization with employees whose capacity to interact with knowledge determines IA process effectiveness and legitimacy, and we use this case to test a multi-level, knowledge-focused capacity framework specific to IA. In what follows, we review the concept of capacity and present the conceptual framework that guides our analysis. We then describe the case study setting and the research methods employed, followed by the results and a discussion of their implications for IA policy and practice.
Conceptual framework
Capacity building (often termed capacity development or strengthening) is a longstanding and complex endeavor that remains ambiguous in practice, despite the variety of disciplines working to develop the concept and its application. The concept of “capacity” is broadly defined as the evolving combination of competencies (skills), capabilities (resources), and relationships at multiple levels (individual, organizational, network) that enables a system to exist, adapt, and function, which interact to enhance or limit overall capacity (
Marsh and Smith 2000;
Brinkerhoff and Morgan 2010;
Howlett and Ramesh 2015). Recognizing the breadth and ambiguity of the concept,
Potter and Brough (2004, p. 337) suggested that it is “as diagnostically useful to say, ‘there is a need for capacity building’ as to say, ‘this patient is unwell’.” A solid grounding for the concept of capacity therefore remains elusive in scholarship and application, with numerous calls for further clarification and refinement for specific contexts looking to solve capacity challenges, including IA (
Harrow 2001;
Howitt and Suchet-Pearson 2006;
Boyd et al. 2013;
Darling et al. 2018). Recognizing the multi-faceted, multi-scalar, and inter-related nature of the different types of capacity is of particular importance when conceptual frameworks are applied across disciplines and institutional contexts (
Harrow 2001;
Gadsby 2011;
Nuyens and McKee 2005).
Interrelated capacities
Across academic disciplines, scholars have identified and categorized different types of capacity to try to add clarity for capacity-building efforts and address specific societal objectives (
Darling et al. 2018;
van der Molen 2018). In environmental governance literature, the most widely discussed types of capacity relate to “adaptive capacity”—associated with the ability of a group to be resilient to changes in their environment (i.e.,
Gupta et al. 2010); “governance and community capacity”—the ability of a group to make decisions and function (e.g.,
Laverack and Labonte 2000;
Matarrita-Cascante et al. 2017); and “policy and institutional capacity”—the ability to provide legitimacy and support for governance functions in a group (e.g.,
Woodhill 2010;
Krishnaveni and Sujatha 2013;
Ramesh et al. 2016), which were drawn together by
van der Molen (2018) under headings of adaptive, integrative, and regulatory capacity in environmental governance. Despite increasing emphasis on the need and intent to accommodate and consider different knowledge systems in environmental governance, little research has developed the knowledge dimensions of capacity, known elsewhere as research capacity (
Natcher et al. 2005;
Howitt and Suchet-Pearson 2006;
Emerson and Baldwin 2019;
Gustafsson et al. 2020;
Spagnuolo 2011).
Research capacity relates to the other supporting capacities that influence the function of environmental governance mechanisms and is often identified as a sub-capacity that supports national and regional development (
Velho 2004;
Chan et al. 2005;
Al-Roubaie 2010;
Andrews et al. 2011). For example,
Howlett and Ramesh (2015) and
Howlett et al. (2017) outlined the dimensions of policy capacity that relate directly to knowledge as “knowledge system capacity.” When governance mechanisms, such as IA, rely heavily on evidence and knowledge, the research capacity of boundary spanning organizations can have an important influence on network-level outcomes (
Marsh and Smith 2000;
Velho 2004;
Howlett and Ramesh 2015). Within IA networks, assessment boards interact intensively with a variety of knowledge sources, which requires certain competencies and capabilities particular to the IA context (e.g.,
Partidario and Sheate 2013;
Bond et al. 2018a;
Maag et al. 2018). Assessment boards generally gather input beyond the technical specifications of a proposed development project, including public comments and local and Indigenous knowledge. This requires a high degree of both individual and organizational research capacity, with direct implications for the performance of IA processes. As a result, there is a need to better understand the dimensions of research capacity in IA bodies tasked with the coordination and interpretation of supporting evidence and knowledge to inform best practices (
Kirchhoff 2006;
van Loon et al. 2010;
Scott 2011;
Bond et al. 2018b;
Loomis and Dziedzic 2018)
Evaluating capacity in IA
Evaluating the state of capacity in any field relies on conceptual frameworks often applied across disciplines to identify potential gaps in individual or organizational abilities and then develop strategies to fill those gaps through interventions (
Boyd et al. 2013). Generic capacity frameworks face the challenge of conceptual ambiguity and vagueness, leading to mismatched or imprecise goals and minimal intervention success when applied across widely varying contexts (e.g.,
Harrow 2001;
Biesta et al. 2011;
Gadsby 2011;
Boyd et al. 2013;
Nuyens and McKee 2005). These generalized capacity evaluations have already been applied to environmental IA systems, identifying larger systemic issues in developing countries where national governments are largely in control of IA processes and the process itself is often not well established (
Doberstein 2001;
Kirchhoff 2006;
Kolhoff et al. 2009;
van Loon et al. 2010;
Kolhoff et al. 2018;
Khosravi and Jha-Thakur 2019). Specialized frameworks have concentrated on individual or organizational levels of capacity, but rarely connect the two (e.g.,
Gupta et al. 2010;
Hamel and Schrecker 2011;
Gustafsson et al. 2020). The spectrum between nuanced qualitative elements, such as mentorship, and highly specific quantitative components, such as financial support, makes balanced evaluations rare (
Fig. 1). The result is a lopsided emphasis on the more easily measurable elements of capacity (
Cooke 2005;
Bates et al. 2006;
Howlett and Oliphant 2010;
Armstrong et al. 2013;
Kislov et al. 2014;
Maag et al. 2018). In Canada, there is also growing emphasis on the acknowledgement of pluralism in decision-making that will require a better understanding of the qualitative dimensions of capacity in individuals and organizations, such as sense of place or sanctity (
Udofia et al. 2017;
Arsenault et al. 2019). As a result, there is a clear need for a refined understandings of capacities particular to IA and how they interact.
We reviewed and compared 24 capacity-specific conceptual frameworks drawn from six disciplines, with an emphasis on those specific to environment and research capacity, to identify common elements and then customize for IA in northern Canada (
Appendix A). Using pattern coding, we identified common characteristics between frequently used and cited capacity frameworks (
Appendix A). When the principles of IA are considered, the suitability of these foundational aspects of capacity for application in IA is limited to the standard operational aspects, as their interpretations tend to leave dimensions of capacity particular to IA unacknowledged (
Boyd et al. 2013). To refine these characteristics, we looked to the foundational principles behind IA, where context, meaningful consideration, and participation are key elements (
Natcher et al. 2005;
Noble and Hanna 2015;
Udofia et al. 2017). We identified three broad themes to incorporate both common and IA-specific capacity constraints: infrastructure, administrative support, and knowledge demand. These higher order themes are supported by sub-capacities that are also interrelated, where more obvious components, like financial and human resources, co-exist alongside more distinctively IA components, such as disciplinary versatility and contextual understanding. In what follows we examine these themes in more detail using a case study of the IA assessment board in the Yukon Territory, Canada.
Study setting
The Yukon Territory rests at the apex of the Rocky Mountains at the junction between the Canadian Shield and the Pacific tectonic plate, which allows for a concentration and wide variety of geological formations, and therefore unique resource opportunities, in close proximity to each other. The first inhabitants of the area, now known as the Yukon First Nations, arrived ∼24,000 years before present (
Bourgeon et al. 2017). They remain a driving force behind the political and economic evolution of the territory, making up ∼23% of the 41,352 people in 2019 (
YG 2020). Located in the northwest corner of Canada, between Alaska, Northwest Territories, and British Columbia, the Yukon shares peoples, cultures, and language groups with other modern jurisdictions.
Land claims and self-government agreements, established under the context of an overarching
Umbrella Final Agreement in 1990, form the fabric and foundation of Yukon governance mechanisms, including interactions between the territorial and federal governments. These agreements look to establish greater local control over regional social and economic development by addressing topics such as wildlife, lands, heritage, and resource management (
UFA 1993). Chapter 12 of the modern-day treaty agreement establishes the impact assessment process (referred to in the agreement as development assessment) and outlines the connections between regional and territorial advisory and review boards as well as connections with complementary processes such as land use planning. The result is a federally legislated decentralized impact assessment process under the 2003
Yukon Environmental and Socio-Economic Assessment Act that supersedes other similar processes, with jurisdiction over all development projects undertaken in the territory (
Government of Canada 2003). The specialized assessment body YESAB was subsequently established in 2006. Assessors are embedded in local contexts by dividing the territory into six districts with local Designated Offices (DO) assessing small-scale projects. In YESAB, both Assessment Officers (AO) and Managers of the Designated Office (MDO) are active assessors and share similar responsibilities. Larger development projects of a certain scale, complexity and political consequence are assessed by the Executive Committee (ExComm) office and are usually large multi-year projects with potential for territory-wide social and economic impacts.
Results
There are many ways to organize the different dimensions of capacity for analysis, but previous frameworks limit their acknowledgement of the interconnectedness of the different elements and their qualitative aspects (
Boyd et al. 2013). Often one aspect of capacity (research or otherwise) at the individual level is directly related to and impacted by organizational- or network-level limitations (
Howlett and Ramesh 2015). The following analysis does not aim to be a comprehensive evaluation of the assessment body, but rather focuses on the strengths and challenges as identified by key informants, using the guiding elements of IA-specific capacity illustrated earlier. Interconnections between different dimensions and with other capacities are acknowledged where possible and explored where necessary, but for clarity and concision we focus primarily on the dimensions presented in
Table 1: infrastructure, administrative support, and knowledge demand.
Figure 2 illustrates these dimensions along the qualitative/quantitative spectrum. Similarly, the analysis prioritizes challenges, if only to offer potential solutions, even while there are well-established areas at YESAB that can be drawn out as examples of strong research capacity.
Infrastructure
We use the term infrastructure to describe dimensions of capacity that are primarily organizational that support research capacity in the IA context. This includes themes of external and internal coordination, as well as perceptions of legitimacy to operate. Coordination of any kind is a shifting challenge across scales and actors, as resources and connections fluctuate and as perceptions of legitimacy, including “acceptable” evidence, evolve. These components require adequate policy and procedural capacity, as well as knowledge management structures, to be fully realized.
External coordination
External coordination for YESAB is multi-faceted, approached through both physical and digital infrastructure that relies on capacity at both the individual and organizational levels. YESAB has a decentralized model for their physical infrastructure, where DOs are established in communities central to the district with the intent of embedding assessors in the local context and facilitating external coordination and outreach. The decentralized model introduces both advantages and challenges into YESAB’s overall operations. For example, situating assessors in communities does make them more accessible to the local community and the proximal First Nation, which facilitates the maintenance of external relationships through established lines of communication and outreach activities. The reliance on face-to-face and informal communication with outside organizations makes DOs the most viable option for maintaining the local area network and ensuring the local connections are made and maintained. Participant 1214 described the advantages of being embedded in a community:
“There’s so many local keepers of knowledge that to have the [Designated Office] in that area aids in information sharing…. It’s nice to just be in the community and be known and people are more comfortable sharing that information.”
However, maintaining a presence in smaller communities has limitations associated with external factors, such as housing availability or long-term employment for spouses, which leads to challenges with filling assessor positions. Participant 0901 summarized:
“We have staff that have lived out of tents and in the back of their cars. We do also get complaints sometimes about ‘you’re hiring all these new people right out of school with no experience’ and frankly, sometimes, I feel like saying, ‘Yes, we’re lucky we got anybody’.”
The external factors were seen as limiting the number of applicants for available positions, the candidates that eventually accept the positions and how long they stay, and were considered particularly relevant in the more rural communities.
Generally, participants felt that YESAB had limited interaction with the public beyond infrequent public meetings and the comments received in the official “Seeking Views and Information” phase of assessments. Participants described this passive approach to outreach as being a challenge, where interactions with the public are mainly through individuals coming into the DO, formal public open meetings, open houses, and notifications for public comment posted at the grocery store or post offices. On a few occasions, YESAB has offered short courses introducing the assessment legislation and process have been presented to First Nations governments and the public, but these one-time events are based on the stability, availability, and the motivation of individual assessors. Participant 1248 described the challenge of communicating with the public regarding their comments:
“A lot of the time, people will have these comments, and we can’t take them into account for whatever reason because of the act or a technicality, and there’s no way to really interact with those people to let them know, and so the learning opportunity is not there.”
Many participants felt that increased outreach might help with concerns around legitimacy as well by clarifying the role and mandate of YESAB and improving the quality and usability of submitted comments.
On a related note, comments received from different actors in the IA network are often based on the work of consultants. This can lead to what participant 1006 termed the “dueling consultant problem,” where First Nations and proponents both hire consultants to address the same topics and present opposing conclusions which then translates to difficulties for assessors in terms of interpretation. As participant 1006 explained:
“There’s no linking together and saying, ‘Okay, how can we get not only the better bang for our buck? Are we asking the same questions? Are we actually talking about apples instead of apples and oranges?’ ”
YESAB was seen as potentially holding a unique position to contribute to the facilitation of resource coordination between external parties while also benefitting from this developed functionality.
Another aspect of external coordination is the dissemination of information and knowledge related to project applications. The YESAB Online Registry (YOR) is an online database of all past and current applications for development projects along with the supporting correspondence, information and knowledge, and the public comments submitted for consideration as part of the assessment. One main intent behind the YOR is to make all information and knowledge associated with assessments publicly available to enhance the transparency of the process, but limitations have been identified. As of 2019, YESAB was working to resolve aspects of this acknowledged shortcoming, confirmed by Participant 0901, “We’re in a process of creating a new online registry which will allow us some better search functions, things like that.” In response to public and proponent feedback, YESAB updated the YOR to increase accessibility in 2019. Past variations of the YOR were criticized as inaccessible (i.e., incompatible file formats, awkward search capabilities), which limited cross-application between similar projects. With the heavy reliance on previous assessments and the associated information and knowledge possessed by assessors and employees, participants viewed this upgrade positively, with the potential to increase knowledge exchange, as well as better supporting cumulative effects assessment.
One exception to the view that more knowledge exchange is better is with respect to Traditional Knowledge, which often contains sensitive information such as the location of sacred sites and traditional hunting grounds. These types of submissions have specific confidentiality policies and procedures outlined in the legislation to maintain a respectful approach to interactions with First Nations cultural heritage. Participants discussed perceived tension with proponents linked to limiting access to sensitive information, who have expressed frustration in their lack of access to many of the participants.
Internal coordination
Participants at all levels of the organization identified internal coordination, mainly centered on data management, as an ongoing struggle, both between and within levels. Participant 1439 summarized these challenges:
“Often I’ll be working on policy issues, and I’ll go to do some word searches on our ‘G’ drive, and I’ll be like ‘holy crap. This is what we’re trying to fix. We already did this in 2015 […] We had a bunch of meetings, and this was the outcome, but it hasn’t been built into our current [operations].’ ”
More established respondents generally felt that time was often spent revisiting solutions that have already been considered and discarded, which they felt led to wasted time, energy, and resources.
Better internal knowledge management has the potential to facilitate internal policy development, maintain external relationships, and support knowledge mobilization. The digital and administrative infrastructure required to support policy work was generally seen as lacking, as summarized by participant 1302: “YESAB hasn’t been very good at documenting operational practice, and that’s really what people need to know to do their jobs.” The organization added a policy-specific position in 2011 and another in 2016, a move that some participants saw as a positive partial solution, though assessors still maintain policy responsibilities alongside their assessment obligations. The other major challenge concerning internal knowledge management was identified by participants as staff turnover, particularly when knowledgeable assessors exit positions or take leaves of absence. Numerous participants identified that an ideal part of training new assessors would involve some overlap with the exiting assessors so that the incumbent could provide context and introductions to the major actors in a community, but this is often not possible. Participant 1248 described the need for more formalized knowledge management: “it’s more of a hope that the knowledge has been transferred enough… I don’t think that people are being tapped as resources effectively when they’re leaving.” Mentorship around the generic components of the assessment process can come from other offices, but without the local overlap, previously established relationships are lost. This interruption in connection between organizations can lead to wasted resources, as initiatives or projects proceeding without adequate contextualization can revisit previously identified problems that went undocumented. Participant 1214_1 articulated:
“Every time someone leaves, there is a lot of information that leaves with them and the history, because that doesn’t get captured anywhere, the history of why we do things the way we do.”
Respondents also described a reliance on previous assessments as a major knowledge source, with limited new knowledge being introduced over the course of new assessments. Access to new knowledge was seen as restricted by the amount of dedicated time available to assessors for seeking new relevant resources and the availability of “outside” knowledge. As previously identified, the internal knowledge management system that does exist was seen as having limited utility in the day-to-day operations of assessors. For example, a small internal repository of useful resources relies on individual assessors for contributions. Participant 1248 remarked on the state of the repository, “Our library needs to be fed into. It’s not really being fed into that much.” Participants reported having little dedicated time to identify pertinent and up-to-date resources. Instead, there was a reliance on previously completed assessments to provide the basic resources for assessments and training. There was also a reported tendency to rely on maintained informal connections to former assessors. Participant 1248 pointed out that this is not a guaranteed resource by saying, “I think there’s this weird idea that when the people are gone, you can call them for information.” In a sense, there is an assumption that the corporate knowledge is not lost but has transferred elsewhere within the broad informal IA network and is therefore still accessible to the assessment board. Many former assessors have moved on to positions in the Yukon Government or one of the First Nations governments, so this assumption is not completely false. However, an over-reliance on accessing previous employees potentially diminishes the impetus to develop appropriate corporate knowledge systems.
Another concern relates to the public availability of academic studies that are housed behind publisher paywalls. In the mandate of YESAB, it is explicit that all information used in assessments should be publicly available, which leaves academically published research in a questionable position. Participant 1355:
“A potential issue around that that’s never really been tested is to what level are we as an organization, as assessors allowed to review and use third-party information like that, because the basis for our assessments is that the information is public and publicly available and transparent.”
With a large portion of academic literature held behind paywalls and therefore unavailable for public consumption, the argument could be made that this should limit how much those knowledge sources should be engaged. The perceived result is a reliance on previous assessments and comments for the majority of the knowledge supporting assessment recommendations.
Financial resources
Participants identified financial resources as a strength of YESAB, with assessors being well supported to pursue professional development and attend meetings and conferences. The assessment body receives full financial support from the Government of Canada, as part of the
Umbrella Final Agreement, administered by the Yukon Government. Accessing training and professional development funds is a common element in nearly all capacity frameworks, and YESAB appears to have the financial resources to support their employees. However, financial resources alone can’t guarantee functionality, as participant 0901 explained:
“You can’t just throw money and more staff at it, because well there aren’t really more staff out there and there’s no housing. You have to continue to look for more longer-term sustainable solutions, and that usually comes down to how you work.”
Legitimacy
Questions of legitimacy were seen as an ongoing challenge for YESAB. Numerous participants identified the perception of neutrality (or lack thereof) as both an external and internal challenge that has ramifications for the legitimacy of the process. External perceptions of bias were discussed from the perspectives of proponents, specialized interest groups, and First Nations governments. As participant 1006 articulated:
“If they [the public, proponents, First Nations] think that staff is undertrained, inexperienced, biased, or using technical information or research to further their own goals, then that’s not helping the organization at all.”
Some of these viewpoints can be managed by establishing and reiterating a clearer understanding of roles and responsibilities of YESAB and more specifically assessors. Externally, transparency has been established by making assessment documents publicly accessible. Questions of consistency between assessors and DOs have been addressed by developing and publishing organizational methodologies and procedures for approaching the various phases of the IA process. Concerns around assessments completed in nonlocal DOs have been raised and are acknowledged by YESAB as an ongoing concern, which they are looking to address through increasing transparency and providing deeper explanations around methodology.
Ensuring assessors maintain a clear understanding of their role in the context of IA was seen as essential to maintaining organizational legitimacy in the eyes of the public. One participant identified the perception of bias as being particularly challenging in the early days of IA, when industry had the impression that assessors held a decidedly environmental protectionist approach to development.
This challenge has led to the development of public documents and internal discussions about maintaining the external perception of neutrality around YESAB and the IA process. Participant 1006 continued: “The organization tries to, certainly through ongoing conversations and training, say ‘This is supposed to be independent, unbiased, review of the facts.’ ” This position was echoed by Participant 1302 in the context of informal connections that contribute to “…trust that conclusions are being made that are unbiased and fair, based on relevant information.” However, the position of neutrality also introduces some hesitancy, as described by Participant 1439: “YESAB is supposed to be neutral and unbiased, maybe that plays into a little bit into concerns about doing too much outreach or meeting with certain YESAB participants.” This conceptualization of neutrality also does not acknowledge the conventional understandings of evidence, facts, and knowledge.
Knowledge demand
Participants reported a suite of demands being placed on the organization and the individual assessors directly related to knowledge. The most apparent was the demand for technical expertise around environmental constraints, thresholds, and regulations, followed by project management skills. Under-acknowledged in both implementation of assessments (as identified by participants) and in research capacity frameworks is versatility—necessary in a multidisciplinary context—and contextual understanding—particular to research capacity in IA. A combination of organizational and individual-level research capacity is therefore required to meet the knowledge demands placed on an assessment board.
Technical expertise
The major knowledge demand placed on the assessment board, from an organizational perspective, is to maintain a suite of assessors with the collective technical expertise to broadly understand the specifications of any application that is submitted, along with an understanding of the contextual factors. This need can create difficulties when intersected with other challenges associated with filling and maintaining assessor positions, such as housing and living in remote communities. The majority of assessors who participated in our research reported having science-related backgrounds that lend well to the environmental aspects of assessments. However, the skills and experience needed to interact with the more qualitative and Indigenous knowledge aspects of IA were seen by participants as needing further development. The result is an unintentional focus on the environmental aspects of assessments, according to participant 1302:
“I think the environmental components often overshadow the social and economic unless the projects are occurring in a really, really sensitive area or if it’s going to have really major impacts.”
The focus generally remains on more quantitative aspects of assessments with rare exceptions.
From the individual perspective, the assessors reported focusing on generalizable skills, with the ability to interpret the knowledge and information presented to them. Participant 1302 explained:
“We’re not experts and we don’t know what areas were used historically for cultural purposes, for example. That information, the cultural impacts, the social impacts on First Nation communities for projects – We just don’t generate that internally … we try to go local and get it, but there’s no experts within YESAB who are generally qualified to speak on those issues.”
The reliance on local organizations and the local public as the source for contextual knowledge regarding potential impacts can be problematic, especially when local organizations experience their own research capacity challenges. Similarly, public input is directly related to the amount of public awareness about the process, ongoing projects, and their role in the process, so the level of outreach organized by DOs can influence how much the public interacts with the IA process.
Project management
Participants identified individual assessors as being responsible for coordinating the acquisition, documentation, and eventual interpretation of the information and knowledge for the assessments assigned to them. This requires more generic skills associated with project management, including timeline management and general administrative tasks such as coordinating meetings and outreach that are the basis for relationship building. In particular, the relationship-building and maintenance aspect was seen as central to the assessor position. Participant 1302 explained:
“I think the real skill that people need to have working for YESAB is the ability to generate and maintain informal relationships because there’s really not a lot of formal opportunities for information sharing and relationship building outside of a specific project.”
This was a particular concern when internal or external staff turnover occurs without overlap, as those relationships then need to start anew.
Versatility (multidisciplinarity)
Another important skill that assessors were seen as needing to develop related to versatility, meaning the ability to fluidly interact with both quantitative and qualitative data from across disciplines, skills that have been under-acknowledged both in IA and in research capacity contexts. Participants referred to themselves as generalists, or knowledge brokers, and that this role comes with challenges that can be met by developing an individual’s versatility. Assessors are required to develop, with minimal direct training, the ability to engage and interpret different forms of knowledge, such as health and social aspects of a project, as well as perform qualitative analysis of the public and interest group comments to identify the major valued components. Participant 1038 acknowledged the challenge: “we want to call it [IA] science but it’s like an art… And it’s a real struggle because it’s a very different form of learning from the traditional [forms].” Some assessors have approached this challenge by attending workshops led by First Nations and similar offerings from other sources that introduce different worldviews and ways of identifying community values. There appears to be room to expand the spaces for assessors to explore their own biases and develop versatility both between disciplines and worldviews that would facilitate their interactions with, and interpretations of, qualitative data and Indigenous knowledge beyond understanding the context of land claims and self-governance.
Another underacknowledged skill connected to disciplinary versatility was the ability to communicate in diverse ways with diverse audiences, including clear argumentation alongside layperson interactions. The interpretation of evidence was seen as fundamental to assessments and transparency around the process, as recommendations are founded on the information and knowledge submitted for consideration. As expressed by participant 1355: “What we always have to come back to as assessors is: based on the information you have, is your conclusion supported through your argument? If it is, then that’s fine.” In the same instance, the assessor is also the public face of the assessment board in the community, fulfilling an outreach role. As expressed by participant 1733: “We are expected to communicate with external stakeholders, with organizations and we are expected to maintain these relationships, and that’s explicit in both [AO and MDO] job descriptions.” The ability to shift between the formal writing and argumentation and informal community interactions is an important knowledge brokering skill that individual assessors develop to support YESAB’s research capacity.
Contextual understanding
The physical infrastructure of YESAB is designed to embed assessors in the community in closest proximity to the projects being assessed. The intent behind this structure is to provide space for assessors to connect with more thoroughly and better understand local values and potential impacts of projects. Participant 1355 identified the (rare) assessor who has returned to the community after leaving for post-secondary education as “…good ones to have because they know the communities, they’re part of the communities, they would like to stay part of the communities.” Tension was reported between the level of personal involvement assessors want in their communities and not overwhelming community organizations with contact. Participant 0901 framed it as a potential criticism either way: “You always hear this; you should be more involved in the community but […] they also don’t want you bugging them all the time.” Assessors who strike this balance are often more successful at building and maintaining community relationships, but many assessors also acknowledged that this requires concentrated individual effort with varied success.
The challenges around contextual understanding connect back to infrastructure, workload, and turnover. Community infrastructure in the smaller communities, most notably housing, exemplifies challenges felt in other dimensions, where contextual understanding is fostered through living in a community but limited by the ability to maintain a presence in the community. Workload is a similar impediment, as participant 1_1439 explained:
“… we have the busy assessment season that can inhibit our assessors’ time to contribute to going out and doing YESAB one-on-ones or attending conferences…I think that’s a barrier for generating and sharing knowledge.”
A related challenge connects back to the previously raised issues associated staff turnover, which also limits contextual knowledge transfer.
Administrative support
Administrative support refers to the organizational and individual abilities to contribute to day-to-day organizational functions. Factors related to this aspect include reliance on mentorship as the main training mechanism and opportunities for professional development as well as for career advancement within the organization. Dedicated time to access new knowledge and the development of an organizational culture of knowledge exchange are also part of the administrative support aspect.
Mentorship
Mentorship was reported to play an important role in YESAB, acting as the primary training mechanism for new employees, including assessors. When a new assessor is hired, the main avenue for training the recruit is through mentorship under the guidance of an MDO. The type and amount of training received is highly variable and heavily dependent on the availability and management experience of the individual MDO. Participant 1302 explained:
“… it’s just an ad hoc approach to how they’re being trained. Some people are just getting thrown into the fire. Some people it’s more methodical and they’re starting at the beginning, looking at the context and how it [YESAB] was created. Others are just going right into doing assessments and just learning as they go. Not a very consistent approach.”
Relying on mentorship as the primary orientation mechanism was seen as introducing inconsistency across the organization and the development of individual assessors, which is then perpetuated as those assessors go on to mentor others, as pointed out by Participant 0301_1439: “If there were issues in how those [previous assessments] were done, they can carry into people’s future practice.” With such reliance on mentorship for maintaining operational continuity, support for mentorship work as part of a position’s responsibilities becomes especially important. Participant 1449 articulated: “We have a lot of trouble, I think, with getting people up to snuff in terms of really understanding what it is they’re doing…. There is quite little support for staff in training other staff, at present.” When training and mentorship are combined with their other duties, including completing their own assessments, reviewing all assessments administered by their office and potentially policy development work, the ability of MDOs to fulfill all these roles becomes strained. As participant 1038 pointed out: “…if someone is floundering, I’m not sure that... there’s anybody to catch them.” Without organizational support or formalized training, turnover becomes more likely. Many participants identified that the effort and time for an assessor to become fully and independently functional is extensive, so to lose a potential long-term employee due to a lack of initial and ongoing support was viewed as being costly to the organization and mission as a whole.
Professional development opportunities
Organizationally, YESAB is well-structured to provide support for training opportunities to bolster individual expertise, such as through workshops, meetings, short courses, and conferences. Participant 1248 noted: “they’re really doing a good job of pushing different training opportunities, and there’s a budget for it.” However, it is on the individual assessors to self-identify gaps in their knowledge-base and seek out the means to fill those gaps. Participant 1214_2 pointed out limitations around accessibility: “the HR department has all the training that people have gone to, but me as an employee, I can’t see what the people have done, … so I can’t ask them questions.” While there are mechanisms for sharing back to colleagues after external training experiences (i.e., training reports), these were characterized as limited in their availability and utility. The main avenues for capturing the information gained through individual training was presenting back to the main group of assessors and completing a short form that is then stored on a shared drive. Some participants expressed frustration with this, such as participant 1248: “…it’s not useful to be growing people individually if they’re not sharing it when they come back.” The connection to internal dissemination and knowledge sharing becomes important, as the opportunities afforded to employees in terms of outside training could have broader collective utility in support of overall research capacity.
Providing opportunities for employees goes beyond training and professional development, particularly the opportunity for career advancement and individual challenge. YESAB is a relatively small organization with a flat hierarchical structure, so there is limited room for an employee to advance professionally in their communities and beyond. For example, in a DO, there are only three levels of employment: administrative assistant, AO, and MDO. Participant 1214_1 explained: “If there’s already a manager that’s there for a long time, there is no opportunity to move up unless you move out or go to Whitehorse and then become an ExComm assessor ...” The prospect of staying in the assessor position with little chance for promotion left some participants feeling as though they became stagnant in terms of career development and cited this as a reason for seeking out other opportunities. The importance of having upward mobility was underlined by Participant 1038: “I looked around and I had no mentors… ultimately, I felt like this was it, it was me…. I plateaued. There’s no place else to go.” Orienting professional development as part of organizational culture supports the development of a challenging, yet supportive environment, and is also connected to the mentorship aspect.
Dedicated time
A dimension that underlies most of the other dimensions is the idea of dedicated time for the required aspects of the various positions. In other contexts, this would refer to time that is strictly reserved for research activities. In the context of IA, this would refer to time for expanding individual and organizational knowledge bases, explicitly built into job descriptions and implementation, but perhaps lacking because of operational constraints. Participants identified that mentorship, external relationship-building, and policy work were built into their job descriptions. For example, participant 1449 remarked: “…on my annual performance review, there’s a line about advancing our understanding of assessment and internal capacity…. Most senior staff are working on these projects.” However, limited dedicated time is afforded to the development, documentation, and upkeep of guidance for new and returning assessors, which extends time spent on orientation. Much of this limitation stems from the volume of assessments that a limited number of assessors are completing annually, which then limits the time available to devote to nonassessment work. Participant 1439 expressed: “I don’t think we always have a ton of time to do self-reflection and [evaluation report] audits,” while participant 1214_2 pointed out that “usually, it’s your manager who is your source of guidance.” Without dedicated time to develop broad guidance documents, there are few mechanisms other than having AOs going to their MDOs directly for guidance.
Organizational culture
Many of the reviewed capacity evaluation frameworks reference the development of an organizational culture that leads to information and knowledge exchange that then fosters creativity and problem solving, along with supporting the development of organizational knowledge. Overall, participants agreed that YESAB fosters such an environment, where the main force behind internal exchange was asking questions, but there were few formal processes and limited documentation. Participant 1439 articulated:
“YESAB has a pretty open-door policy. All the managers, if you have questions or you need to sit down and work through something with someone, they’re usually pretty available and don’t always have the answer but give you some direction and some key points to consider in making your decision.”
This idea connects to the dimension of mentorship, where the availability of managers impacts the culture of exchange, making support for individual mentors central to how the organization functions. The other aspect is fostering an environment where assessors feel supported and comfortable interacting with their colleagues and supervisors. Participant 1248 described the environment as: “you have different tangents that you can go on that people will follow you, rabbit holes that I think will be really helpful.” These explorations help develop both the individual assessor’s ability to perform assessments and organizational knowledge, as different issues are brought to the forefront for discussion and multiple perspectives on said issue can be explored collectively.
The converse of this dimension is that the need to ask questions is compounded by limited documented guidance for assessors. Participants identified that the process for documenting internal procedures were limited and underutilized, which can lead to inconsistencies, misunderstandings, and increased demand for training and mentorship. As participant 1439 described: “I’ve heard clearly from day one that there’s a lack of guidance. Even if people go away for a year […] and come back, it’s like what’s changed?” Organizational culture is central to operationalizing other dimensions, as witnessed by other connections drawn in preceding sections, but in the context of YESAB have been found to rely on individual connections, motivation, and knowledge for cohesion across assessors and employees.
Discussion
In examining the interrelated capacities affecting YESAB, the main assessment body in the Yukon Territory, a variety of challenges related to general capacity building appear to have been met, while others particular to IA and societal goals of pluralism have arisen. This observation leads us to identify some insights we consider relevant to IA more broadly. The interaction between individual capacity and organizational capacity are intertwined in a way that makes developing research capacity a complex balance between investing in individuals and investing in organizational supports. This finding supports the previous research on capacity building generally, where multi-scalar development is considered a necessity to establish long-lasting sustainable change (e.g.,
Velho 2004). For example, without improved internal knowledge management infrastructure, the individual dimension of “disciplinary versatility” we have identified, known elsewhere as “close to practice” (
Cooke 2005) or “evidence management skills” (
Kislov et al. 2014), would not easily contribute to organizational knowledge. Similarly, organizational culture oriented towards the continuous acquisition of new knowledge, including facilitating the pursuit of pluralism by engaging multiple knowledge systems, is only useful in so far as the individual assessors participate. In the case of Yukon Territory, reliance on past assessments for access to relevant information and knowledge is being perpetuated by perceived necessity, as workload constraints diminish the time available for assessors to seek out new approaches, information, and knowledge. The idea of “dedicated time” is captured in research capacity frameworks to support the acquisition of new information and knowledge (e.g.,
Hamel and Schrecker 2011;
Maag et al. 2018). Building organizational infrastructure to support “dedicated time” can improve “disciplinary versatility” by increasing the diversity of knowledge being accessed during an assessment. Our findings also highlight that assessors are regularly being exposed to complementary knowledge systems beyond more accepted and familiar technical considerations, which can contribute to external perceptions of legitimacy and boundary spanning (
Bond et al. 2018a).
“Contextual understanding” is central to the consideration of multiple knowledge systems, since IA is inherently situated in a project’s environmental, social, and economic context locally, regionally, and nationally (
Scott 2011). The intent of YESAB’s decentralized organizational structure is to embed assessors in local contexts so that they will be better placed to understand local community issues. This strategy contributes to overall assumptions that a process guaranteeing the inclusion of Indigenous and local knowledge will be more equitable (
Kamarck 2007;
Pollitt 2003;
Hughes 2010;
Glucker et al. 2013;
Emerson and Baldwin 2019), and (or) less colonial (
Howitt and Suchet-Pearson 2006). Due to the small size of both the assessment board itself and the communities in which assessors are situated, the research capacity of individuals will be central to the effectiveness and legitimacy of the IA process, as the assessors become the local “face” of the organization.
Kirchhoff (2006), in a framework specific to IA capacity, refers to relationship-building and maintenance aspects of an assessor’s work as “network and linkages” while
Cooke (2005) referenced a more general research capacity framework of “linkages and partnerships.” These aspects are also variants of what
Howlett and Ramesh (2015) call “organizational political capacity,” in this case referring to the status of the assessment board amongst the network of organizations working towards development decisions. Not only are assessors tasked with maintaining the assessment board’s presence in local communities, they are also performing the actual assessments, underlining their role as a knowledge broker (
Maag et al. 2018) and the need to develop their individual ability to interact with multiple knowledge systems. Here, administrative supports can help assessors establish and maintain external relationships, including initial orientation and ensuring adequate time for relationship maintenance and the development of individual contextual understanding.
We found that assessors are individually responsible for maintaining their individual technical and nontechnical expertise, but the growth and diversification of individual skills are reliant on available time, their academic and professional backgrounds, their exposure to and understanding of the specific context within which they are working, and their understanding of the concept of pluralism and multiple knowledge systems. Most capacity frameworks refer directly to the individual skills of practitioners and the ongoing development of those skills through the organization (
Doberstein 2001;
Cooke 2005;
Hamel and Schrecker 2011). Individual and organizational capacity is therefore interlocked and reciprocal, particularly in contexts with high staff turnover and where contextual understanding is considered central to the organizational mandate. While keeping administration informed by practice is an ideal, resources are necessary to support organizational evolution and knowledge management (
Howlett and Ramesh 2015). Our results suggest that individual assessors can contribute to the development of YESAB policy and procedures to a limited extent, with most of their time being spent on the main mandate of completing assessments. This factor may have slowed the evolution of internal organizational policy, which our participants identified as limiting the overall research capacity of the assessment board. A partial solution has been the expansion of the employee base to include policy analysts to support and guide overall organizational development, though assessors are still involved.
As IA processes move towards full inclusion of Indigenous Peoples and complementary worldviews in decision-making processes, lessons can be drawn from the research capacity challenges facing YESAB. Overall, YESAB has built broad internal research capacity sufficient to operationalize IA legislation on a daily basis, though there are spaces requiring organizational attention. They have built a culture of information and knowledge sharing, with spaces for interaction between individuals and designated offices. However, this informal system has translated into a reliance on individuals for knowledge management in lieu of more fully developed organizational mechanisms. Notably, there appears to be a heavy reliance on mentorship for initial training and orientation of new assessors, where participants identified inconsistencies between DOs. The IA-specific capacity framework presented in this paper offers a useful starting point for assessment bodies and assessors in contexts advocating for the effective implementation of pluralism in IA.