Science-Metrix provided the bibliometric line of evidence in support of the Evaluation of the NSERC Discovery Research Program. This study was conducted in two phases: a pilot study examining efficient and cost-effective ways to disambiguate researcher names for the 12,500 researchers included in the scope of the study and the bibliometric analysis itself. The broader evaluation was conducted by Prairie Research Associates (PRA).

Science-Metrix’s pilot study aimed to assess the feasibility of relying on the automated disambiguation of author profiles in Scopus, rather than having to manually construct them, with the aim of achieving robust conclusions in an evaluation context. This was intended to reduce costs while maintaining the reliability of bibliometric findings. The pilot study recommended various strategies for making use of Scopus unique author identifiers (AUIDs), depending on the size of the population being investigated.

NSERC requested two major lines of inquiry for the bibliometric line of evidence: (1) a characterization of overall performance levels; and (2) an assessment of the effects of NSERC Discovery funding. The first entailed comparing the bibliometric performance of successful and unsuccessful applicants across several categories: funding programs, evaluation groups, provinces, institution sizes, career stages, gender, funding bins and quality bins (i.e., peer ratings). Included in the benchmarks were the provincial and national averages. The second inquiry entailed advanced econometric modelling on a large population of 12,500 researchers to trace the influence of NSERC Discovery funding on their career trajectory, looking at several evaluation dimensions (e.g., publication output, citation impact, international collaboration and interdisciplinarity). The researchers were tracked over the 2009–2017 period and several control variables (e.g., individual- and year-fixed effects, years since PhD, quality of application) were integrated in the models to enable the interpretation of observed associations as causal or not. The project fed into a larger mixed-method evaluation run by NSERC’s Evaluation Division in partnership with PRA.

Read the final report of the complete evaluation here [PDF].

Read a conference paper presenting the pilot study and its recommendations here.

For more information about NSERC, see here.

Image credit: iStock Photo