Evaluation of the Minerals and Metals Sector (MMS) Science and Technology (S&T) (2009)
Client: Natural Resources Canada (NRCan)
Description: This evaluation assessed the science and technology (S&T) programs of the Material Technology Laboratory (MTL) and the Mineral and Mining Science Laboratories (MMSL). MTL and MMSL focus on the application of sound science and the development of innovative technologies for the needs of government, industry and other key stakeholders.
This evaluation examined three issues: (1) relevance and rationale; (2) results and success; and (3) cost effectiveness (including design and delivery issues) and alternatives to the delivery of the laboratories’ S&T activities.
The evaluation was based on evidence from: (1) a document and file review; (2) two client satisfaction and web outreach surveys; (3) interviews with key laboratory staff, clients, partners and other stakeholder organizations; (4) detailed project reviews based on project documents and interviews with project managers and clients; and (5) a review of S&T outputs including a bibliometric analysis of scientific publications.
Download the report
[HTML] – Evaluation Report and Management Response
Evaluation of the CFIA’s Enhanced Bovine Spongiform Encephalopathy (BSE) Initiative (2009)
Client: Canadian Food Inspection Agency (CFIA)
Description: In response to the first BSE cases in May of 2003, the Government of Canada, through Health Canada and the CFIA, enhanced its existing BSE-related activities and rapidly developed a comprehensive suite of science-based measures to effectively minimize the likelihood of exposure, amplification and spread of BSE within the cattle population. Since 2003, the CFIA obtained additional financial resources for the Enhanced BSE Initiative, including the following components:
- Enhanced Surveillance and Testing
- Enhanced Tracking, Tracing and Enforcement
- Expansion of Domestic Slaughter Capacity
- Removal of Specified Risk Material
- Expanding Export Markets
This evaluation examined the following issues: success and achievements, governance, design and delivery, relevance and continued need, performance measurement and reporting, and cost-effectiveness and alternatives.
The evaluation was based on evidence from: (1) interviews with key informants internal and external to the CFIA and (2) a review of documents and files from the CFIA and relevant external organizations.
Download the report
[HTML] – Evaluation Report
[HTML] – Management Response
Evaluation of the Explosives Safety and Security Branch (ESSB) of the Minerals and Metals Sector (MMS) (2010)
Client: Natural Resources Canada (NRCan)
Description: Natural Resources Canada (NRCan) mandated Science-Metrix to carry out the evaluation of the Evaluation of the Explosives Safety and Security Branch (ESSB) of the Minerals and Metals Sector. This evaluation assesses the programs and major activities carried out, individually and in collaboration, by the Branch’s two component organizations, the Explosives Regulatory Division (ERD) and the Canadian Explosives Research Laboratory (CERL), from fiscal years 2004-05 to 2008-09. The ESSB was created as part the renewal of the Minerals and Metals Sector. The Branch and its current organizational structure have been in place since 2007-08. Before this, CERL and ERD were part of the Minerals Technology Branch (MTB). This constitutes the first evaluation of the ESSB.
This evaluation examined five issues related to relevance and to performance. Four methods were used in this evaluation: (1) a review of external literature (international and national) and a comparative analysis of explosive regulatory and S&T systems in other countries; (2) interviews conducted with ESSB staff and with Canadian and international stakeholders; (3) case studies based on the study of 14 projects/activities across the five main activity areas of ESSB and including additional interviews with ESSB staff as well as clients and partners; and (4) an extensive review of over 200 internal files and documents.
Multiple lines of evidence establish that the ESSB has been successful in fulfilling its mandate and achieving its main objectives throughout the period encompassed by the evaluation. The overall effectiveness of ERD’s regulatory oversight and CERL’s S&T outputs was high: ERD successfully administered the Explosive Act & Regulations (EA&R) and developed and validated regulations and policies on the basis of CERL’s scientific input. CERL also made significant contributions to external clients and stakeholders through the transfer of knowledge, expert advice and technology. These activities ensured a strong and credible framework that improved public safety, strengthened national security and enhanced industry efficiency.
Download the report
[HTML] – Evaluation Report and Management Response
Evaluation of the Collaborative Research and Development Grants (2010)
Client: Natural Sciences and Engineering Research Council of Canada (NSERC)
NSERC mandated Science-Metrix to carry out the evaluation of its CRD program. The CRD program is intended to give companies that operate from a Canadian base access to the unique knowledge, expertise, and educational resources available at Canadian postsecondary institutions and to train students in essential technical skills required by industry. The mutually beneficial collaborations are expected to result in industrial and/or economic benefits to Canada. CRD Grants support well-defined projects undertaken by university researchers and their private-sector partners. Direct project costs are shared by the industrial partner(s) and NSERC. Projects may range from one year to five years in duration, but most awards are for two or three years.
This evaluation, which covered the 1997–2008 period, included several data collection methods: 1) a documents/ literature, files and program data review, 2) a grants review, 3) key informant interviews, 4) five web surveys (conducted with academic researchers, industrial partners, unfunded academic researchers, unfunded industrial partners, and highly qualified personnel (HQP)), 5) an economic impact analysis, and 6) six case studies.
The findings indicate that the CRD program is relevant, well designed, appropriately delivered, and generally provides considerable long-term benefits to industrial partners, academic researchers and HQP.
Download the report
[PDF] - Evaluation Report
[PDF] - Management Response
Evaluation of SSHRC Research/Creation Grants in Fine Arts Program (2008)
Client: Social Sciences and Humanities Research Council of Canada (SSHRC)
Description: SSHRC is a federal funding agency that promotes and supports university-based research and training in the social sciences and humanities. Based on growing evidence that university-based artists, despite increasing in number, are underrepresented in federal funding programs, SSHRC in 2002 approved the development of a three-year pilot program to support these artist-researchers in their pursuit of artistic knowledge.
The formative evaluation of the Research/Creation Grants in Fine Arts Program examined the following issues: (1) relevance; (2) design and delivery; (3) outputs and immediate outcomes; and (4) risks and opportunities.
The evaluation was based on the triangulation of evidence obtained with the following methodological instruments: (1) a comprehensive review of files and documents related to the program; (2) a roundtable workshop with grant recipients; and (3) web surveys of three groups: funded applicants, unfunded applicants (comparable group), and post-secondary managers and research or grant officers.
Download the report
[PDF] – Evaluation Report
[PDF] – Methodological Appendices
[PDF] – Management Response
Evaluation of the Networks of Centres of Excellence New Initiative (NCE-NI) (2009)
Client: Social Sciences and Humanities Research Council of Canada (SSHRC)
Description: The NCE–NI pilot was established in 2003 by the NCE Secretariat to: (1) facilitate the creation of networks; (2) support networking activities among well-established researchers or research teams to encourage them to develop new partnerships with receptor communities; and (3) respond to the needs for interaction, partnership, and networking.
This evaluation examined the following issues: (1) the results of the pilot, in terms of outputs and outcomes; (2) the delivery of the pilot, as delivered by the NCE Secretariat and by the NCE–NI networks themselves; and (3) the relevance of the pilot.
Multiple methods and lines of evidence were used to address the evaluation issues: (1) a review of administrative documents, files, data and web-based sources; (2) case studies of the five funded NCE–NIs (including interviews and a documentation review); (3) a comparative analysis with comparable programs; and (4) a counterfactual analysis.
Download the report
[PDF] – Evaluation Report
[PDF] – Management Response
Online Survey Administration to Support the International Blue Ribbon Panel Assessment of Peer-Review Practices at the SSHRC
Client: Social Sciences and Humanities Research Council of Canada (SSHRC)
Description: Among the key strategic priorities for 2008-2011 noted in the document Framing our Direction, SSHRC had undertaken a Blue Ribbon Panel assessment of the quality and integrity of its peer-review practices to enrich the quality of its peer-review practices. In order to gauge the opinion of the research community in this regard and to collect their comments, SSHRC mandated Science-Metrix to administrate an online survey of SSH faculty in Canadian universities for the Blue Ribbon Panel in charge of the assessment.
Nearly 20,000 researchers were invited to respond to the survey. Participation in the survey was high, with a response rate of over 35%. More than 6,000 survey questionnaires were completed and analyzed using quantitative and qualitative analysis tools. The survey results confirmed the value and credibility of SSHRC’s peer-review system. Survey results also provided substantive support for a large number of recommendations made to SSHRC by the Panel.
Download the report
[PDF] – Blue Ribbon Panel Report (Promoting Excellence in Research – An International Blue Ribbon Panel Assessment of Peer Review Practices at the SSHRC)
