Lessons learned from the CADTH Evaluation

Published on March 31 2017

The Canadian Agency for Drugs and Technologies in Health (CADTH) has just released its latest evaluation report. The Science-Metrix Evaluation Services team conducted this extensive evaluation from September 2015 to December 2016. Several findings from the evaluation relate to data, including issues concerning data collection, access to data and the linkages that can be made between different data sources. These findings informed one of the evaluation’s three recommendations. In a data-driven world, they can also provide valuable lessons of benefit to our broader clientele.

The CADTH evaluation focused on the relevance and performance of the Agency for the years 2012–2013 to 2015–2016. It aimed to identify those impacts to which CADTH contributed through its role as a producer or broker of health knowledge and information. It also assessed CADTH’s current and potential role in light of observed trends in both the Canadian health care system and health care worldwide. Six technical reports summarizing the collected data were produced during the 16-month evaluation and shared with CADTH management and Health Canada.

The evaluation tried to gather data on long-term outcomes such as social, economic, and health or health system benefits. It also looked for examples of changes in policy practices or enhanced policy coordination resulting from CADTH’s services and products. A key challenge for the evaluation was finding evidence to link these longer-term impacts back to CADTH, chiefly due to gaps in the available data. Based on the information collected and feedback from multiple stakeholders, the evaluation determined that these data were not available in a form that made such linkages possible. As noted in the report, “This evaluation found there was no centralized repository or system for storage, retrieval, processing, and quality assurance of performance data at CADTH. […] while individual units collect data and report on performance, there is no consistent way to make the links between the data and performance indicators as per CADTH’s [Performance Management] Strategy.”

But with new data sources and technological advancement come new opportunities to collect and connect data. The recommendation stemming from these findings was that CADTH should improve its performance measurement to better quantify and qualify its impact on the health system and its contribution to downstream impacts. This recommendation has implications for the quality and consistency of data collection both across an organization and over time, as well as the linkages that can be made between datasets at the federal and provincial levels in order to clearly attribute such impacts. The evaluation noted that such data gathering could be undertaken in collaboration with CADTH’s funders, such as Health Canada and the Provinces, because of their enhanced access to health outcomes and health care systems indicator data.

In addition to a broader application of the findings and recommendation above, other important lessons were learned by the evaluation team that can be used to inform future evaluations. These also mirror lessons learned from a soon-to-be-released Data Mining study Science-Metrix has completed for the European Commission. First, the design of an evaluation of an institution or program with a pan-Canadian mandate would benefit from an initial stakeholder mapping exercise to understand the landscape, federal/provincial relations and the positioning of the institution or program within a complex web of stakeholders, partners and funders. Second, while implementation processes, customer engagement and governance may be client priorities, summative or end-of-funding cycle evaluations should maintain a strong accountability for results focus. And third, if value-for-money is an evaluation criterion then the evaluation design should allocate sufficient resources to focus on the achievement of outcomes, especially long-term outcomes given the inevitable need for secondary and primary data collection with provincial and territorial stakeholders.

Identifying data gaps and planning to fill them is an ongoing process. Capitalizing on existing data sources, seeing the potential applications of new sources and considering how these can be linked to efficiently track outcomes will enrich future evaluation findings and broaden the evidence base for decision-making.

The final report of the Evaluation of CADTH is available here: [PDF]

The CADTH Management Response is available here: [PDF]

About CADTH:

CADTH was created in 1989 by Canada’s federal, provincial and territorial governments, who believed that Canada needed a coordinated approach to assessing health technologies. The result was an organization that harnesses Canadian expertise from every region and produces evidence-informed solutions that benefit patients in jurisdictions across the country.

CADTH is an independent, not-for-profit organization responsible for providing health care decision-makers with objective evidence to help make informed decisions about the optimal use of health technologies, including drugs, diagnostic tests, and medical, dental and surgical devices and procedures. In addition to evidence, CADTH also provides advice, recommendations and tools.


Image: iStock Photo