Key Element 8: Demonstrate Accountability for Health Outcomes

A population health approach focuses on health outcomes and determining the degree of change that can be attributed to interventions.

What is accountability for health outcomes?

Accountability in the past focused much more on inputs (resources utilized), processes (activities) and outputs (products). Within a population health approach, a much greater emphasis is placed on accountability for health outcomes and determining the degree of change that can actually be attributed to interventions.

Changes of interest include health status, the determinants of health inequities, knowledge, awareness and behaviour, shifts in social, economic and environmental conditions, as well as changes to public policy and health infrastructure.

The identification and measurement of short- and medium-term outcomes are as important as long-term outcomes to ensure ongoing relevance for stakeholders whose agendas are shorter-term. Longer-term outcome evaluations include process evaluations to determine whether a policy or program is reaching its target population and impact evaluations to measure the immediate results of a program or policy.

Regular and timely reporting of evaluation results and outcomes for all stakeholders is also an important part of accountability.

Resources to Increase Understanding:

What does it mean to be accountable for health outcomes?

• Evaluation in Health Promotion: Principles and Perspectives – World Health Organization
• Defining Accountability – Public Health Agency of Canada

Key questions

• What will be measured?
• How will it be measured?
• How will results be shared?

A) What will be measured?

Accountability requires the capacity to measure performance against key commitments on an ongoing basis through evaluation.

A sound performance measurement strategy should cover:

• The main activities of the program (what will be done?)
• The clients or target populations (who will benefit?)
• The expected results (what will be achieved?)
• The performance indicators (how will we objectively know?)
• The data sources (where will we get the information?)
• The methodological considerations (how will we measure and analyze, and at what costs?)

A population health approach must include health goals, targets, outcome indicators and baseline data collection

• “Health goals” is used in a generic sense and refers to a planning framework that typically includes a clear breakdown of the strategies to be undertaken and the parties which will be responsible.
• Targets are quantified statements of the level/magnitude, direction and timing of the desired change. They define what will constitute success.
• Outcome indicators are statistical measures which measure progress towards the desired change.
• Baseline measures represent current statistical levels on health-related indicators.

B) How will it be measured?

Accountability tools such as health impact assessments support increased accountability for actions and investments in health. They do this by setting forth criteria for determining the impact of current or proposed policies and programs.

Health impact assessments have the benefit of a consistent process across sectors to look at health impacts; this benefits those who would not normally consider how their decisions affect health. However, unless this process is enforced in some high profile way (e.g., by the Auditor General), it is not likely to be effective.

C) How will results be shared?

Public reporting of its impact to the wide range of stakeholders is a cornerstone of the population health approach. It is part of the commitment to engage the public and partners. Creative attention-getting communication strategies may be needed. Examples include:

• population health “report cards” at national, provincial and regional/municipal levels
• cataloguing and sharing “best practices”
• documentation of case studies that show precisely how population health initiatives are undertaken and maintained.

Reporting may also target special population groups, especially those facing particular risk factors or conditions.

How is accountability for health outcomes achieved?

Accountability efforts may be strengthened by following the advice of the WHO European Working Group on Health Promotion Evaluation (1999). In “Health Promotion Evaluation: Recommendations to Policymakers,” the following guidelines are laid out:

• Encourage participatory approaches to evaluation that support meaningful involvement by those with an interest in or who are potentially affected by the initiative.
• Agree upon a set-aside percentage of total financial resources to support evaluation costs.
• Support the use of multiple evaluation methods that allow for the assessment of both process and outcome information.
• Fund research on appropriate approaches and tools for population health evaluation.
• Establish a training and education infrastructure to develop expertise in the evaluation of population health initiatives.
• Create and support forums and networking opportunities for information sharing on the challenges and methods of population health evaluation.

8.1 Construct a results-based accountability framework

A results-based accountability framework establishes clear performance measures at the outset of programs. It includes the following:

• a clear statement of the roles and responsibilities of the main partners involved in delivering the policy, program or initiative (e.g. a sound governance structure)
• a clear and logical design that ties resources to expected results (e.g. a results-based logic model)
• an outline of the performance measurement strategy, including costs and key indicators that will be tracked
• the schedule of major evaluation work
• an outline of the reporting provisions to ensure transparency to stakeholders who have an interest (and duty) to be informed on program progress (or lack thereof).

Main partners should agree to the accountability framework early on.

8.2 Determine baseline measures and set targets for health improvement

The international literature reveals five possible approaches to setting targets. They are:

• (a) historical comparisons, where the targets are set using the levels of an indicator from the past and the present to create reasonable future estimates for that indicator
• (b) normative comparisons, where the targets are set based on the level of performance on an indicator achieved by similar interventions in comparable settings (these are often referred to as benchmarks)
• (c) theoretical standards, where the targets are based on what research and theoretical evidence suggests is reasonably achievable
• (d) ideal or absolute standards, where the targets are complete eradications of the problems;
• (e) compromise standards, where the targets are set through some combination of the other four methods.

The choice of which target-setting method to use depends on the current state of knowledge and the availability and accessibility of comparable data sets.

Target setting, especially relating to health determinants, is difficult. However, efforts to overcome the challenges are worth the potential benefits, and data on the progress toward targets is often instrumental in advancing policy, program planning, evaluation, and resource reallocation. That is one of the key reasons why baseline measures are employed, as they allow for tracking of progress over time.

8.3 Institutionalize effective evaluation systems

Effective evaluation systems require that five elements be consistently in place:

1. There must be sufficient time to plan, implement, and finalize all required evaluation steps.
2. There must be adequate budget, staff, expertise and equipment.
3. There must be explicit and clear agreements from stakeholders regarding the key evaluation questions and the standards of a satisfactory evaluation.
4. Data must be available and accessible.
5. Decisions must be made in a timely and effective manner.

8.4 Promote the use of health impact assessment tools

Health impact assessments can provide comprehensive and compelling information.

8.5 Publicly report results

A good reporting strategy builds upon a variety of information products. Comprehensive and detailed reports are useful for some stakeholders, but shorter and less technical versions (including one-page fact sheets and executive summaries) are better for other stakeholders.

Text, tables and graphs, as well as visuals, can be combined to effectively convey key points.

A variety of communication methods is best. For instance, presentations and mailings raise awareness and interest, short sessions or focused papers build understanding, and consultations and follow-ups support applications of the findings.

Resources that Build Capacity:

What frameworks or guides can help create a meaningful intervention evaluation?

• Monitoring and Evaluation – The World Bank
• The Good Indicators Guide: Understanding how to use and choose indicators – Association of Public Health Observatories
• Community Development Evaluation Storymap and Legend – Neighbourworks Training Institute

Examples Illustrating Application:

What are some examples of well-evaluated interventions?

• National Evaluation of the Community Action Program for Children – Public Health Agency of Canada