We compiled a list of over 2300 publications arising from the CNBC and its predecessors under AECL primarily from 160 progress reports of the AECL Physics Division from 1949 to 1996 archived by the Canadian Nuclear Laboratories library, and from annual activity reports from the CNBC for 1997–2017.
The AECL reports contained official publication lists and conference talks for each Branch. In the reports up to 1959, the neutron papers had to be distinguished manually from those of other physics fields. In 1960, neutron scattering, neutron nuclear physics, and neutron detector development were consolidated into a single branch of AECL, making the task of finding the neutron papers easier. The lists from the Theoretical Physics Branch were also examined for papers related to the theory of neutron scattering.
Finally, each paper’s bibliographic entry was cross-checked with the record on the journal’s website to check for errors and to obtain the digital object identifiers or other links to the publications online.
Citations are frequently used as proxy measurements of scientific impact, or quality, of the papers. Proper benchmarks are key to the interpretation of citation-based indicators, however. For this reason, the CNBC commissioned Science-Metrix Inc. to analyze the citations relative to the world averages in the sub-fields in which the CNBC papers appear. The Science-Metrix study also compared the results with that of five major neutron beam facilities in the United States and France and with three other major Canadian research facilities that provide users with access to probes for materials research (X-rays, muons, and electrons at the Canadian Light Source, TRIUMF, and the Brockhouse Institute for Materials Research, respectively).
The international neutron facilities were chosen for their size, reputation, and location. The High-Flux Isotope Reactor (HFIR, USA) and Laboratoire Léon Brillouin (LLB, France) are reactor sources with a mid-sized complement of instruments and scientific histories comparable to the CNBC. The Los Alamos Neutron Science Center (LANSCE, USA) is a spallation source of comparable infrastructure size for neutron scattering, with some additional scientific missions. The NIST (National Institute for Standards and Technology) Center for Neutron Research (NCNR, USA) is the North American pre-eminent reactor-based facility, running a robust program and whose location in Maryland fostered frequent collaboration with the CNBC personnel. The new Spallation Neutron Source at Oak Ridge National Lab was not chosen, because we desired facilities with publications over many years. Finally the Institut Laue-Langevin (ILL, France) is the global flagship facility as the brightest reactor source with the most beamlines and publications of any neutron beam facility.
This particular collection of comparison facilities we believe should have similar level of biases and gaps in the bibliometric data, including comparable levels of missing publications and citations, misattributed authorship, and nonindexed specialty publications.
4.1. Publication data set
The Science-Metrix study was limited to papers published since 1980, because of incompleteness in the citation data in the Web of Science (Clarivate Analytics) database for earlier years.
Publications from the comparator facilities were identified primarily if the name of the facility was found in the author affiliations, the abstract (since 1991), or the acknowledgements (since 2008). For the HFIR, the study relied on the publication database posted on the Oak Ridge National Laboratory website. For most of the comparator facilities, these methods failed to identify papers for the whole period. The Canadian Light Source is a new facility, which officially opened in 2004. Some of others have had names changes or changes in how authors have identified their affiliations over time. For example, the name “Canadian Neutron Beam Centre” dates back only to 2005, and for decades the authors from the CNBC’s predecessor branches of AECL often listed their affiliation simply with AECL without reference to their branch. Such practices would have make it difficult to identify the papers from the CNBC prior to 2005 had we not compiled the full publication list manually.
All papers found from each facility were included. For example, research using muon beams to study materials at TRIUMF were not distinguished from TRIUMF’s larger body of research focused on nuclear and particle physics.
Figures 1 and
2 show the publications identified by Science-Metrix for each facility over time and serve primarily to illustrate the data set on which the citation analysis was performed.
From this publication data and numbers of neutron beamlines at each facility, it can be estimated that the CNBC’s publication intensity, measured by number of papers per beamline per year, has been in the low end of the range of publication intensities for this group of neutron facilities in recent years. This finding contrasts with an international peer review in 2004 that found the CNBC to have a publication intensity that was competitive with the most prominent neutron beam facilities in the world. The same review, however, warned that investment in the beamlines and staff at the CNBC had not been keeping pace with other neutron facilities, thereby predicting a relative decline in productivity, which we now observe here; although the CNBC maintained a steady output of research over time, other facilities have increased their productivity significantly.
4.2. Scientific impact metrics
The four measures of scientific impact calculated were the average of relative citations (ARC), the share of highly cited papers (HCP10%), the Citation Distribution Index (CDI), and the average of relative impact factors (ARIF).
In the calculation of all of these quality metrics, citation rates are normalized according to the world average for each subfield and year, thereby producing a relative citation score for each paper or journal. There is no subfield for neutron scattering in Science-Metrix’s journal-based classification system. Rather, since neutron beams are applied to a broad range of problems, research using neutron beams is published in many different journals, which are in turn classified into many different subfields. Some primary subfields include: general physics, biophysics, chemical physics, inorganic and nuclear chemistry, materials, and applied physics. Thus, relative citations are determined in comparison to the citation rates of such topical fields rather than against other papers that used a neutron scatttering technique. Furthermore, the citations are more often made by experts in these broadly defined fields who are interested in the materials being studied than by those mainly interested in the methods used.
ARC is the mean of the relative citation scores of all papers from each facility in a given year. By definition, the world average of relative citations equals 1.
As the mean of the distribution of relative citations can be significantly increased by a few highly cited papers, another common indicator of impact is an institution’s ability to produce such “hit” papers. HCP10% is the fraction of an entities’ papers that was among the top decile of all papers in their subfields in that year. By definition, the world average of HCP10% is 10%.
A limitation of HCP
10% as an indicator is that it only uses the citations from the top decile of papers. The CDI is a measure that represents the full shape of the distribution by using the relative citations from all 10 deciles of papers as inputs. The CDI is higher if an entities’ papers are underrepresented in the low deciles or over-represented in the high deciles. By definition, the world average of CDI equals 0. In theory, the range of CDI goes from −50, the case of all papers being in the lowest decile of relative citations, to 50, the case of all papers being in the highest decile of relative citations. In practice, −25 or +25 are very low or high scores that actually occur. See
Campbell et al. (2016) for more information onthe CDI (where the CDI is referred to as the “Relative Integration Score”).
ARIF is the average impact factor of the journals the papers are published in. As with the other metrics, ARIF is normalized by year and subfield. By definition, the world ARIF equals 1. ARIF is different from the preceding metrics because it is indicative of perceived scientific value at the time of publication, rather than the subsequent impact, which is indicated through citations of the paper itself.
These four impact metrics were calculated for each facility over the range of available data and are summarized in
Table 1. To estimate uncertainties, Science-Metrix took many samples on the data to observe variance between the samples and produce confidence intervals. These intervals are not formal estimates of error, because the results in
Table 1 are exact calculations on all of the available data. Rather they illustrate the range that is expected to be observed if many new publications are added in the future and if these new publications follow the same distribution of citation scores. For example, if the further publications were to arise from the CNBC that followed the same pattern as seen from the CNBC over 2000–2017, ARC scores ranging from 1.19 to 1.53 would be expected 95% of the time. The intervals narrow with increasing numbers of publications. The same confidence interval for the ILL, which has the largest numbers of papers and relatively stable results over time (shown in section 4.2.1), has a width of only 0.06.
A key observation from this data is that the publications arising from all these facilities are performing at or above world average on all four indicators, and most are performing well above average (e.g., ARC > 1.3). These results suggest that having access to unique national shared resources for materials research, including neutron beams, enhances scientific impact.
The performance of the CNBC’s publications were competitive with those of most of the comparator facilities. The NCNR in the United States was the top neutron beam facility in all four indicators, while the two neutron beam facilities based in France (ILL, LLB) were both at the bottom. This results for the ILL seem surprising since it has been the preeminent neutron facility globally for decades and it maintains a very high reputation. Further examination of the data and citation trends in France compared with Canada and the United States did not reveal a clear explanation for the difference. The most likely explanation is that the very large number of papers arising from the ILL, across a wide breadth of subject areas, mask the highly cited publications in the averages. To the best of our knowledge, there is no similar published institutional analysis of the ILL. These results may be also impacted by relative subject-area strengths and sociological factors, such as, language, scientific culture, and policy. Specifically, the business models and mandates of ILL and LLB differ somewhat from that of the North American facilities.
Science-Metrix also examined the performance of the CNBC publications within individual subfields, according to its journal classification system. The CNBC papers were found to fit into 30 of the 176 research subfields, whereas 80% of the papers fit into just six subfields. The three subfields making the top contribution to citation scores were applied physics, materials, and biophysics, as shown in
Table 2. Other subfields in which CNBC publications were frequently found included general physics, materials, chemical physics, inorganic and nuclear chemistry, fluids and plasmas, health sciences, geochemistry and geophysics, mining and metallurgy, mechanical engineering and transport, and polymers.
Science-Metrix examined trends in the five top subfields that are contributing to the results in our paper to determine general trends between United States, Canada, France, United Kingdom, and Germany. Over the period 1980–2017, France scored the lowest ARC of this group 1.2, while Canada at 1.3 scored the same as the United Kingdom at 1.3. Germany scored 1.2, while the United States scored 1.5. Similarly for HCP10%, France was again at the bottom with 12%, Canada at 13%, Germany 13%, United Kingdom 14%, and United States 17%. These results suggest that systematic factors that reduce France’s citation scores in these subfields overall could be dragging down ILL and LLB’s scores, although the effect is not large enough to be conclusive.
4.2.1. Impact metrics over time
To explore the above data on a deeper level, the impact metrics were calculated in 3-year rolling intervals and shown in
Figs. 3–
8. Scores of ARC, CDI, and HCP
10% were not calculated for intervals in which fewer than 30 papers were identified.
Figures 3–
8 show a significant amount of fluctuation from period to period, which indicates some uncertainty in the data. Some of the fluctuation can be attributed to random causes. Indeed, we observe that those facilities with the most publications tend to exhibit lower fluctuations in these quality indicators because more of the random effects are averaged out than for the facilities with fewer publications. The indicator that is calculated from the fewest data points, the HCP
10% data, exhibits the most fluctuation.
Although it is difficult to determine the nonrandom causes, some correlation with historical events affecting the CNBC can be suggested. AECL faced significant budget cuts beginning in the mid-1980s and was mandated to focus increasingly on the nuclear power industry. Although AECL had to cut back on fundamental research, it chose to keep the CNBC, providing stable support throughout the 1980s and early 1990s. Concerns about the future of the CNBC became more acute in a second wave of federal government budget cuts in 1993–1996, ultimately leading to a decision by AECL to eliminate funding for the CNBC in 1996. The negative effect that such concerns undoubtedly had on the remaining staff morale, coupled with the departure of some senior scientists in 1997 and 1998, may explain some of the drop in these indicators, especially visible in the HCP10% data. In contrast, performance dramatically recovered after the CNBC was taken over by the NRC in 1997, which was fully supportive of fundamental research using the CNBC and encouraged pride in operating a national resource for the scientific community. Following this recovery, there was another small performance drop in the mid-2000s. The cohort of senior scientists that had characterized the 1970s, 1980s, and 1990s had mostly retired by this time, and the science staff in the mid-2000s could be described as a mix of mid-career scientists and new hires, many of whom were learning the role of an instrument scientist. After this brief drop, performance returned to near its historic average.
Despite these fluctuations, the general stability of the CNBC’s performance over a 37-year period is perhaps the most striking feature. The business model of the CNBC as a user facility and its ability to attract high quality users may help explain the facility’s resilience, since the quality of its publications over this period have depended on the excellence of the facility’s users. Expert users who maintain their engagement despite turnover in facility staff can serve as a bridge that helps maintain continuity and high scientific impact.
The general trends of ARC and CDI over time for the CNBC are similar with a few contrasting features. The ARC shows a local maximum in 1986–1988, which does not appear in the CDI. Such a difference could be explained by a small number of highly cited papers that have a large impact on the ARC, but have less impact on the CDI. Indeed, two very influential papers by W.J.L. (Bill) Buyers and collaborators appear in these years. They concern experimental confirmation of the Haldane gap (
Buyers et al. 1986) (described earlier in section 3.2) and the discovery of superconducting pairs in an uniaxial antiferromagnet, URu
2Si
2 in collaboration with Riso National Laboratory, Denmark (
Broholm et al. 1987). In addition, Brian Powell and collaborators from the NRC published a highly influential paper on a clathrate hydrate structure (
Ripmeester et al. 1987).
Opposite patterns are seen in the years around 1982 and 1994, in which the ARC drops to the world average level, but the CDI remains strong and well above world average. This outcome implies a strong performance overall, despite under-representation in the highest decile of relative citations (as the HCP
10% data in
Fig. 5 shows).
Boh ARCs and CDI are high in the time windows that include the years 1998–2002 (coinciding with the recovery of the CNBC under the NRC as already mentioned). An examination of the papers in these years revealed a large number of highly cited papers. Although CDI is not strongly influenced by a one or two highly cited papers, the CDI will rise if a large portion of the papers are highly cited. Indeed, the HCP
10% data in
Fig. 5 shows that over a quarter of the CNBC’s papers in 1999–2001 were in the top decile.
Among the sample of influential papers in 1998–2002 are six publications with John Greedan
1 (McMaster University) and later joined by Bruce Gaulin (McMaster University) on structures and magnetism of geometrically frustrated antiferromagnets including the spinel LiMn
2O
4 and the pyrochlores Tb
2Ti
2O
7 and Y
2Mo
2O
7 (
Greedan et al. 1998;
Gardner et al. 1999a,
1999b,
2001;
Wills et al. 1999;
Gingras et al. 2000), four publications by John Katsaras and collaboators using small-angle neutron scattering on morphology of phospholipid mixtures and the location of cholesterol in membranes (
Léonard et al. 2001;
Luchette et al. 2001;
Nieh et al. 2001,
2002), and two publications by Thomas Holden and collaborators on measurement of intergranular strains in structural alloys of aluminum or zirconium (
Pang et al. 1998,
1999). More influential papers from these years include a study of stresses in a magnesium–aluminum alloy by John Root and collaborators (
Gharghouri et al. 1999), a study of the high-density amorphous phase of ice by Ian Swainson, Eric Svensson, and collaborators (
Tse et al. 1999), and a study of the relaxor Pb(Mg
1/3Nb
2/3)O
3 by Bill Buyers and collaborators (
Wakimoto et al. 2002).
A second set of years with high ARC, CDI, and HCP
10% is 1990–1992. The most highly cited paper on the CNBC’s publication list was Varley Sears’ tables of neutron scattering lengths and cross-sections from 1992 (
Sears 1992), cited over 2800 times, according to Google Scholar. These years included three more influential studies by Buyers and collaborators on the superconductor URu
2Si
2 (
Isaacs et al. 1990;
Mason et al. 1990;
Broholm et al. 1991) and a study on spin freezing in the geometrically frustrated pyrochlore antiferromagnet Tb
2Mo
2O
7 by Bruce Gaulin, Zin Tun (CNBC), and collaborators (
Gaulin et al. 1992).
In more recent years (2007–2013), the CNBC’s most highly cited papers have been in quantum materials, biophysics, and materials engineering. The quantum materials papers include Lachlan Cranswick and collaborators’ study on superconductivity in LaFe
1−xCo
xAsO (
Sefat et al. 2008); Zahra Yamani, Bill Buyers and, collaborators’ study that found the reason for missing entropy in URu
2Si
2 (
Wiebe et al. 2007); and Zahra Yamani and collaborators’ study of magnetic correlations in multiferroic LuFe
2O
4 (
Christianson et al. 2008). In biophysics, John Katsaras and collaborators published five influential papers on model membranes (
Kuĉerka et al. 2008,
2011;
Marrink et al. 2008;
Pabst et al. 2010;
Heberle et al. 2013), including two notable studies on structural parameters of lipid bilayers, i.e., area per lipid and bilayer thickness, using simultaneous data analysis from both X-rays and neutron experiments (
Kuĉerka et al. 2008,
2011) and a demonstration that cholesterol prefers the interior of polyunsaturated lipid membranes (
Marrink et al. 2008). In materials engineering, Michael Gharghouri and collaborators showed that the difference in yield strength between tension and compression in magnesium 0150 aluminum alloys can be greatly reduced by increasing precipitates (
Jain et al. 2010). Ron Rogge and collaborators studied the role of residual stress in cracking of steel pipelines, providing data that is now used by the oil and gas pipeline industry to better predict pipeline lifetimes (
Van Boven et al. 2007).
4.2.2. International collaborations rates and impact metrics
An institution’s international co-publication rate (ICR) may serve as an indication of scientific demand for its resources. It may also have some utility as a predictor of impact using the hypothesis that higher impact may be expected when scientists seek out the world experts on their topics as collaborators. ICR, however, is heavily impacted by factors such as size of the scientific community in the host country and proximity to potential foreign collaborators. For example, institutions in the United States often have strong performance despite lower ICRs than similar institutions in Canada or European countries. This is simply due to the greater probability in the United States that the best collaborator can be found domestically, because of its large scientific community; this trend is shown in
Fig. 9. The papers from each facility were identified as international collaborations if more than one country was represented in the papers’ author affiliations, and ICR was calculated as the fraction of such papers among all the facility’s papers. The United States’ neutron beam facilities have lower ICRs than the counterparts in France and Canada, despite higher performance in the quality metrics. The ILL consistently had the highest ICR over the period of this study. This high ICR is consistent with its mandate to provide access to the member countries who contribute funds to its operations.
There has been a worldwide trend toward increasing ICR over time, and all the facilities examined in this study exhibited an upward trend. However, the greatest increase in the CNBC’s ICR occurred in the several years following the transfer of CNBC to the NRC in 1997. This post-1997 rise in ICR was consistent with the NRC’s model for the CNBC as a national user facility that acted as Canada’s contribution to a global network of neutron beam facilities.
To quantify the impact of international collaborations, Science-Metrix calculated the average relative citations on the subset of papers that were international collaborations, denoted ARC international, for all the available data in 1980–2017, and shown in
Table 3.
International collaboration helped HFIR’s ARC score, but interestingly, had little or no effect on ARC for the other facilities (ILL, LLB, LANSCE, NCNR). One interpretation of this result is that the users of these other facilities may have already maximized the value that they can get from effective collaboration. For the facilities in France, it is relatively easy to collaborate with other countries, both because of geography and the facilities’ business models that encourage collaboration. For the facilities in the United States, heavily competitive merit-based processes for distributing beam time may strongly encourage the collaborations that will be most effective.
For all of the Canadian facilities, ARC international was higher than the ARC for all papers, consistent with two ideas about how Canada can best engage with, and utilize Major Research Facilities. First, for those facilities hosted in Canada, combining competitive and easy-to-access infrastructure with welcoming and knowledgeable staff is a major attraction for global scientists. For the CNBC, the continuous application and scheduling process was an important part of its reputation for being responsive to scientist’s needs. Second, Canadian scientists, more frequently than American counterparts, need to engage collaborators outside their country to make the biggest impact, a phenomenon that may simply be a function of Canada’s smaller scientific community.