In the midst of the big data revolution, the public and private sectors must be proactive in finding new ways to exploit novel data sources and underutilized administrative data. Within a governmental context, a global transition is presently underway towards establishing a greater role for evidence to play in the policymaking process; in tandem, the processes for gathering, analyzing and interpreting evidence are also undergoing profound changes, with the increasing interest in using data mining and big data analysis techniques as a prime example of these shifts.

Within this context, Science-Metrix undertook a two-year study for the European Commission’s Directorate-General for Research and Innovation (DG RTD). The goals of the project, titled Data mining: Knowledge and technology flows in priority domains within the private sector and between the public and private sectors, were to

1. document the use, best practices, benefits, and limitations of data mining (and big data) in the private and public sectors, with a special emphasis on their application for the evidence-based formulation, implementation and evaluation of policies by governments in the R&I context;
2. develop a stakeholder-validated framework to guide the design and implementation of data mining (and big data) projects in the R&I policy context;
3. validate and improve the framework, derive new knowledge of relevance to R&I policymaking, and assess the potential benefits and limitations of data mining approaches for deriving such information by applying the framework to six case studies on key innovation policy issues for the private sector;
4. put forward recommendations—to be validated at an expert workshop along with the framework—to the DG RTD to guide their future use of data mining approaches.

The study was performed in two sequential phases, each taking place over a period of one year. The steps conducted in the first phase aimed at fulfilling the first two goals of the project. This included a literature review as well as interviews with key informants; these provided the basis for developing a novel framework to shape the conduct of data mining projects to inform R&I policy. This framework was then validated at a workshop bringing together experts in the data mining field, concluding the first half of the project.

The steps taken in the second phase of the project covered the two last objectives listed above. The most important step in this part of the project consisted in the application of this framework to six case studies:

1. Multidisciplinary research and innovation;
2. Patterns of R&I cooperation between public and private sectors; 
3. Open access availability of scientific publications;
4. Use and impact of European Investment and Structure Funds for R&I activities in EU regions;
5. Fast growing enterprises;
6. Use and impacts of tax incentives for R&I by Member States.

Read the report here.

Campbell, D., Tippett, C., Struck, B., Lefebvre, C., Côté, G., St-Louis Lalonde, B., Ventimiglia, A., Roberge, G., & Archambault, É. (2017). Data Mining. Knowledge and technology flows in priority domains within the private sector and between the public and private sectors. Prepared by Science-Metrix for the European Commission. ISBN978-92-79-68029-8; DOI 10.2777/089

Requests for access to the case studies report can be submitted in writing to info@science-metrix.com, subject to approval by the European Commission.

Campbell, D., Tippett, C., Struck, B., Lefebvre, C., Côté, G., Archambault, É. (2017) Data Mining on Key Innovation Policy Issues for the Private Sector: Application Report. Prepared by Science-Metrix for the European Commission.

Image credit: iStock Photo