Public Health Units (PHUs) in Ontario are required to evaluate their programs and services, yet face many challenges doing program evaluation. These challenges show the importance of building evaluation capacity (EC) to support effective program and service delivery.

In 2016 a Locally Driven Collaborative Project (LDCP) titled Building Evaluation Capacity in Ontario’s Public Health Units was initiated. An EC assessment tool was selected and the core project team worked with its creator, Dr. Isabelle Bourgeois, to adapt the tool and assess EC in PHUs. Of the 36 PHUs, 32 completed the tool and participated in interviews about their organizational profile. The results of the assessment showed that most PHUs are in the “Developing” category and have some work to do to improve their EC.

In addition to the assessment, the core project team completed a scoping review of evaluation capacity building (ECB) strategies. Six themes came out of the scoping review:

• Strong Leadership

• Organizational Environment

• Adequate Resources

• External Supports

• Strong Skills and Knowledge

• Comprehensive Organizational Evaluation Framework

These themes appeared important for creating a strong evaluation culture; however, the literature did not include information about their effectiveness.

Following this, the core project team intends to build on the worked to determine effective strategies that will strengthen key elements of EC within public health. This will be done by:

1. Testing out different promising strategies as identified in the scoping review (March 2015) in 10 PHUs based on capacity level and structure using an action research framework

2. Establishing a documentation of practice repository of findings from the action research projects that will assist PHUs in meeting the evaluation requirements of the Foundational Standard

3. Providing PHUs with an opportunity to develop in-house ECB skills

Using an Action Research Approach, the core project team worked closely with participating PHUs to select ECB strategies and to collect information about the implementation and outcomes of those strategies through a variety of methods including progress reports, interviews, and a post-project EC assessment. Findings are summarized in an online repository and accessible to all PHUs.

This project increased knowledge and skills among participating PHUs to do action research and implement ECB strategies. The project also increased knowledge of effective ECB strategies for all 36 PHUs and provided new knowledge on ECB to the broader evaluation community.

Reports

• Scoping Review of Evaluation Capacity Building Strategies

• Summary Report

• Full Report

Published Work

Bourgeois, I, Hotte, N, Simmons, L. Osseni, R. Measuring Evaluation Capacity in Ontario Public Health Units. Canadian Journal of Program Evaluation. 2016.

Public Health Unit Materials

Algoma Public Health

• Logic Model

• Evaluation Hub - Outputs and Outcomes Resources

Brant County Health Unit

• Evaluation Brief 1 - Why should we do evaluation?

• Evaluation Brief 2 - Evaluation Questions

• Evaluation Brief 3 - Stakeholder Engagement

• Evaluation Brief 4 - Determinants of Success in Evaluation part 1

• Evaluation Brief 5 - Determinants of Success in Evaluation part 2

Chatham-Kent Public Health

• Logic Model

• Planning & Evaluation Workshop for Managers

• Determining an Evaluation Vision and Strategies - Agenda

• Pre-workshop Self-assessment Report

• Post-workshop Interview Report

• Using Standard Pig for Logic Models

HKPR District Health Unit

• Logic Model

• Research Policy

• Appendix A - Evaluation Research Proposal Template

• Appendix B - Evaluation and Research Algorithm

• Consent Letter Template

• Feedback Form Template

Hastings Prince Edward Public Health

• Cornell Evaluative Thinking Focus Group Protocol

• Cornell Evaluative Thinking Inventory

• Corness Evaluative Thinking Observational Checklist

• Management Interview Questions

• Post-Workshop Survey

Huron Perth Public Health

• Logic Model

• How To Guide: 6 Thinking Hats

• How To Guide: Cookie Evaluation

• How To Guide: Critical Friend Conversations

• How To Guide: Gibbs Cycle

• How To Guide: Paradigms Exposed

Leeds, Grenville & Lanark District Health Unit

• Logic Model

• Planning & Evaluation Community of Practice (COP) Charter

• Planning & Evaluation COP Focus Group Tool

• Planning & Evaluation COP Minutes Tool

• Planning & Evaluation COP Pre-assessment Tool

North Bay Parry Sound District Health Unit

• Logic Model

• Knowledge Translation Framework

• Pre Survey Tool

• Post Survey Tool

Ottawa Public Health

• Logic Model

• Staff Evaluation Assessment Tool (English)

• Staff Evaluation Assessment Tool (French)

• Evaluation Guidelines

• Evaluation Guidelines Presentation (English)

• Evaluation Guidelines Presentation (French)

Region of Waterloo Public Health

• Logic Model

• Evaluation Report