On evidence-based decision-making by Canada’s protected areas managers: a reply to Irvine et al.

We welcome the opportunity to reply to the Comment provided by Irvine et al. (2021), entitled “A more nuanced analysis of evidence-based decision making by Canada’s protected area managers: a comment on Lemiuex et al. (2021)” (Lemieux et al. 2021a). The focus of the Comment was in response to our recently published article “ ‘Free Fallin’? The decline in evidence-based decision-making by Canada’s protected areas managers” (published in FACETS, 6 May 2021; Lemieux et al. 2021a). The generation of dialogue and discussion around the issues raised in our paper was among our key goals in conducting the two surveys described in the paper and publishing the results and conclusions emanating from them in the first place. Thus, our initial response to the Comment is “mission accomplished” for which we are quite grateful.

Several useful and valid points were made by Irvine et al. (2021) in their Comment, especially related to survey design and administration and discussion on potential nuances associated with the use of evidence. This includes being aware that the use of evidence can differ depending on geographical scale, socio-political context, and management issue. We agree fully with these observations and some of the other nuances identified by the authors within their Comment. Also, Irvine et al. (2021) restate or extend many of our own observations, interpretations of results, and associated recommendations to enhance evidence-based decision-making by Canada’s protected areas managers. These discussions ultimately reinforce the significance of our findings. It is clear that we all have similar objectives: to improve evidence-based decision-making and, by extension, conservation outcomes in Canada. We reply to the claims made by the authors, providing further clarification and evidence where necessary, paying particular attention to the authors’ concerns over methods and interpretation of results. Our reply is organized sequentially as they are raised by the authors in their Comment.

First, on page 1, Irvine et al. (2021) claim that we state that there is a “dramatic decline” in the use of evidence. Language matters, especially when it is employed to characterize the work of others. Not once do we use “dramatic decline” to characterize our results. We base our interpretations on statistical evidence presented within our paper. We specifically use “statistically significant” to describe the declines in the use of all 16 forms evidence by Canada’s protected areas managers between 2013 and 2019. We also use “concerning” to characterize these results. It is our interpretation that when all 16 forms of evidence included in our study reveal a statistically significant decline in use that the results should be characterized as concerning.

Second, Irvine et al. (2021) turn their attention to evidence syntheses, an area of study that several members of their author team have published on, cited in Lemieux et al. (2021a) and referred to within their Comment (see specifically Thomas-Walters et al. 2021). Irvine et al. (2021) note that evidence syntheses were not included as one of the 16 forms of evidence in our survey, and that our results are therefore based on an incomplete view. As the Comment notes, evidence synthesis approaches are becoming more commonplace in the conservation field and over time may become more readily available to protected area managers and decision-makers than they are today. While Canadian-based conservation-related syntheses are scarce at present, we share the hope of Irvine et al. (2021) that this tool be an important contributor to decision-making in the future as it continues its gradual emergence from its current nascency in the conservation field domestically and internationally.

Given that evidence syntheses (especially meta-analyses and systematic reviews) relating specifically to Canadian conservation contexts have only recently become more common and accessible to conservation practitioners, it is our opinion that the omission of evidence syntheses as a source of knowledge in our survey instrument should be considered understandable. This is supported by recent observations made by Thomas-Walters et al. (2021) that revealed that an “… apparent lack of available syntheses for many environmental issues means that use [of environmental syntheses] can be limited and tends to be opportunistic…” (p. 1–2). We also dedicated nearly a half a page of our limited space for discussion specifically on the potential value of evidence syntheses to support decision-making by protected areas managers (see page 652, Lemieux et al. (2021a)). In this instance, Irvine et al. (2021) are not arguing for a position other than that already taken in our article. We take the position that evidence syntheses were more than adequately acknowledged and discussed in our article, especially given the limited space available for discussion in research-focused papers that require space for methods and results.

Our inclusion of 16 different forms of evidence represents a more comprehensive picture of available and use of various forms of evidence than the extant literature and should be considered complementary to the work of Thomas-Walters et al. (2021) (and other conservation-related evidence review advocates such as Fazey et al. (2004) and Cook et al. (2013)). Going forward, a single category of “peer-reviewed sources” of information would be insufficient. We advocate that Canadian evidence syntheses efforts recognize differences among approaches to knowledge synthesis. A distinction must be made, as Thomas-Walters et al. (2021) and others before (Thomas and Harden 2008; Grant and Booth 2009; Cook et al. 2013; Moher et al. 2015) pointed out—that rigor of synthesis methods as well as sources of evidence must be considered in efforts to evaluate evidence synthesis use in decision-making.

Third, Irvine et al. (2021) claim that we present a “hierarchy” that is “faulty at both ends of the spectrum” (page 2). We do not know where this claim by the authors is rooted. Not once do we claim to be presenting a hierarchy in relation to our results. The word “hierarchy” does not appear in the text of our manuscript, anywhere. Our tables simply display the survey results sorted from highest to lowest overall mean values to facilitate interpretation by readers. This is consistent with Lemieux et al. (2018), making comparisons between years and articles more understandable to readers.

Fourth, we note Irvine et al.’s (2021) concern about the percentage of our survey respondents being from Alberta; specifically, they claim that our results may be “presumptuous” and “dubious”. The concerns Irvine et al. (2021) express regarding this needs to be thoughtfully deliberated. We considered the potential effect of the high proportion of Alberta respondents in our initial analysis. When Alberta participants were removed from the sample, there were very few differences in the results (see Tables 1 (use) and 2 (value)). Identical and consistent with our findings pertaining to the total sample and reported in Lemieux et al. (2021a), our analysis revealed that all forms of evidence declined in use when respondents from Alberta were removed. Furthermore, 15 of the 16 forms of evidence included in our study exhibited statistically significant (p ≤ 0.05) declines in use when the 2019 data are compared with 2013 use (Table 1). The only notable differences (when Alberta participants were removed) pertained to overall mean scores and substantiveness of differences (effect sizes), which differed slightly between various evidence types, with some increasing, some decreasing, but most being similar. As such, our results very much hold true across Canada’s protected areas managers with respect to the decline in use of various forms of evidence when Alberta respondents are removed from the analysis.

Similar observations were made when perceived value of evidence was examined. Significant and substantive differences in the valuation of forms of evidence with Albertan participants excluded from the sample is documented in Table 2. When compared with evidence valuation in Lemieux et al. (2021a), which reported responses from across Canada, the forms of evidence most valued by respondents (i.e., staff assessments and legislation) remained the same in both samples. For 12 of the 16 forms of knowledge, declines were observed in how evidence is valued. The largest, significant declines were observed in the valuation of staff assessments and databases between 2013 and 2019, for both the full sample and the Alberta-excluded sample.

Fifth, on page 4, concerning the interpretation of Table 2 (perceived value of various forms of evidence), Irvine et al. (2021) challenge our statement that local knowledge or Indigenous knowledge has increased in value, since Table 2 indicates no significant change in Indigenous knowledge, and a decrease in value of local knowledge (page 645 of original article). The authors add that the value of staff assessments has declined, and evaluation of peer review has increased. In this instance, we concur with Irvine et al. (2021) regarding an error made in one aspect of our interpretation of Table 2. While the data and statistics provided in Table 2 are correct, we stated that “both local and Indigenous forms of knowledge have increased in perceived value since the 2013 survey”. However, we should have stated “Table 2 indicates no significant change in the value of Indigenous knowledge, and a small decrease in value of local knowledge”. We thank the authors for pointing this out, enabling us to correct this interpretation (Lemieux et al 2021b).

Sixth, on page 4, (and relatedly elsewhere), Irvine et al. (2021) claim that “…in 2019 the research team did not collect data on the organizational affiliation of the survey respondents” and “…it is unknown how many of the respondents were from governments and how many were from regional conservation authorities”. We couldn’t agree more that collecting data on respondents’ organizational affiliation is important, which is why this was included as the very first (optional) question for respondents in our survey which is readily available to readers online in our Supplementary Material. Therefore, organizational affiliations were collected, at the discretion of the participant.

We offer the following rationale as to why organizational affiliation was optional in the 2019 survey and not reported on in our article. The primary reason for this decision is reaffirmed in Irvine et al.’s (2021) Comment. The authors underscore the potential of politics to influence the results (page 5 and elsewhere). Organizational affiliation was optional for this reason, the desire to obtain authentic responses that may not be achieved if there is potential for participants’ identities to be revealed (e.g., fear of repercussions within their organization, especially junior staff).

Expanding on this point, at the time the survey was administered, a number of provincial/territorial protected area agencies in Canada were facing a number of diverse policy and management capacity issues, including budgetary and human resource capacity constraints (e.g., Office of the Auditor General of Ontario 2020). Target-based conservation had also become a complex, politicized, policy issue pitting quantity (increasing total area declared protected) versus quality of protection (Lemieux et al. 2019; Botchwey and Cunningham 2021). It was decided that participants should have the power to decide what level of detail they wanted to provide with respect to their organizational affiliation given these concerns.

We were also working in the spirit of a co-production of knowledge approach (Norström et al. 2020), working with practitioners from the onset to support our research design (i.e., to identify priority research questions), to enhance participation amongst members within their own professional community, and to elevate the chances of results uptake to improve evidence-based decision-making. Many provincial/territorial government-protected areas organizations employ a very small number of full-time employees at the head office level. For example, the entire Nova Scotia Protected Areas and Ecosystems Branch has only 12 full-time staff (novascotia.ca/nse/protectedareas/contact.asp), Newfoundland Provincial Parks Division 13 (gov.nl.ca/tcar/contact/tourism-culture-and-parks-branch-contact/), and Northwest Territories Conservation Networking division 10 or fewer. While these numbers do not include staff at the individual park level, where such individuals even exist (beyond visitor services), they are extremely low. It is worth noting here that the Thomas-Walters et al. (2021) paper on evidence syntheses similarly did not report on provincial/territorial participants by their respective organizational affiliations. Like our approach, they are considered collectively at the provincial/territorial level. The significance of the results in both papers should not be diminished based on attempts to protect the workplace well-being of participants (assuming this was Thomas-Walters et al.’s (2021) similar intention). This is a common practice in social science/human subject research (Simsek and Veiga 2001; Warner et al. 2011; Roberts and Allen 2015; Macey and Fink 2020).

That said, the authors’ concerns over nuances associated with more regional or local and issue-specific decision-making contexts are valid. Such an analysis is not possible given our broad, Pan-Canadian approach to sampling. However, it is important to note that at no point in our article do we claim that the results are scalable and generalizable to the regional/local scale or management issue-specific contexts. In fact, we see an opportunity prompted by Irvine et al.’s (2021) observation: our survey (or an adaptation thereof) could be administered in more regional and issue-specific contexts to identify important geo-political and (or) management nuances in the use of evidence in decision-making by Canada’s protected areas managers. We come back to this prospect at the end of this Reply.

Finally, as noted in our Methods on page 642 (Lemieux et al. 2021a), we are clear that participant recruitment was focused on protected area managers. Conservation Authorities in Ontario, who manage Conservation Areas–a land-use designation used only in the province of Ontario—are not protected areas. Such areas are not recognized as protected areas provincially in Ontario nor are they included in Canada’s Protected and Conserved Areas Database (CPCAD) (managed by Environment and Climate Change Canada (Environment and Climate Change Canada (ECCC) 2021). Despite the claim that managers of conservation areas may be included in the study, we are certain that they are not.

In summary, we left it to participants to identify their organization to provide the flexibility to obtain more honest responses, and to enhance anonymity and avoid the potential for workplace repercussions—which is an important and common risk management measure in social science research (Warner et al. 2011; Macey and Fink 2020). These are acceptable procedures related to research involving human subjects, informed by people working in the protected areas sector themselves, through a knowledge co-production approach.

To conclude, we appreciate and respect the work of Irvine et al. (2021) that has focused on advancing evidence-based decision-making in Canada’s conservation sector. There were several insightful comments that we can use to improve future studies. Debate on research design and associated outcomes is encouraged and welcomed in the spirit of advancing knowledge and progress on the state of evidence-based decision-making in all aspects of conservation including protected areas management.

As our Reply has shown, we state confidently that our results should be interpreted by readers as valid. Among other important findings, the results revealed statistically significant declines in all forms of evidence between 2013 and 2019, and the fact that 181 managers from across Canada responded to such a time-demanding survey (with over 160 questions and statements, 85 of which generated data used in Lemieux et al. (2021a), covering a wide range of themes requiring evaluation by respondents), our article represents nothing short of the most comprehensive and current view of the state of evidence-based decision making by Canada’s protected areas managers. It also represents the first examination of the state of evidence-based decision-making by protected areas managers over time, not only in Canada but globally. We stand by these results, the decisions made to ensure participant anonymity, the interpretations contained within, and re-emphasize that the results should be considered concerning by Canada’s diverse and growing conservation community.

Finally, we would like to extend an offer of collaboration to Irvine et al. (2021) as our collective interests and expertise, and commitment to rigorous methods and producing valid results, present an important opportunity to come together to enhance evidence-based decision-making in support of effective biodiversity conservation outcomes in Canada. Indeed, the most important contribution of the Comment is the way it highlights the clear need for further research and related dialogue with the conversation community around how to continuously improve the adoption of evidence-based decision making in protected areas management at all scales and in all parts of our country. We would be happy to work with the Comment authors and the Parks Canada Agency that two of them work for to develop a robust approach to this issue and to the expansion of our survey work to a fulsomely national scale with widespread coverage of protected area managers and program leaders. Such a study could embrace the different forms of protected and conserved areas and identify the extent to which decision-making supports their management either in general terms or in ways that vary with the form or scale of protection at play. Collaborative research of this nature would substantially enhance and support the efforts of a wide range of protected areas practitioners and researchers in a manner that would contribute directly to our deeply shared goal of effectively conserving and protecting Canada’s rich biodiversity and natural heritage.