Guidance to Integrate Results Measures into Performance Audits

Purpose

The Performance Audit Practice is seeking to more consistently integrate measures of results into its performance audits. The purpose of this guidance document is to support audit teams to understand the objective of integrating results measures into performance audits, and to better integrate results measures into the planning, conduct, and reporting of performance audits.

Background

The mission of the Office of the Auditor General of Canada is contribute to well-managed and accountable government for Canadians. Well-managed government implements programs and services that deliver their intended results for Canadians in an efficient and effective manner. Accountable government identifies and measures its intended results, and communicates to Parliament and Canadians the extent to which intended results are being achieved.

In order for the Office to assess whether it has contributed to well-managed and accountable government, it needs to know the extent to which intended results are being achieved. By identifying the intended results of the programs and services audited and reporting on their status, and then monitoring those results through audit work at some point in the future, the Office is able to communicate about the achievement of results in areas where it has conducted audits. Accordingly, the Office is working to implement a new indicator in its performance measurement framework: the percentage of measures of departmental program results included in its audit reports where the Office has determined that results have improved.

Role of performance audits

For the performance audit practice, achievement of the Office’s mission begins with the planning of each performance audit. Once the audit topic has been chosen through the strategic audit planning or other process, the auditor identifies the result(s) that the program or activity being audited is intended to deliver or influence. The focus should be on results of importance to Canadians. The auditor then identifies what results are, or could be, measured; in other words, what measures are, or could be, used to assess whether the program has achieved its intended results? Again, the focus should be on identifying results measures that would be meaningful to stakeholder or client groups and ideally, to Canadians generally.

An important part of the objective of the Office’s direct engagements is to identify significant problems in the administration of federal programs, services, and activities that need to be addressed; and to make recommendations for improvement that will help ensure the delivery of intended results for Canadians. In addition to making observations and recommendations about management and administration of programs and activities, the Office wants to make observations on results measures. This may include making observations on the quality of measures, the quality of information systems, or the implementation of the Policy on Results; the quality of reporting on results achieved; and making recommendations to improve results measures or information systems.

Planning the performance audit

The attention that the Office is giving to results measures is taking place within the context of the Treasury Board’s Policy on Results. As part of the planning of the audit, the audit team should familiarize itself with the Policy, and with the entity’s implementation of the Policy.

Having identified the topic for the audit, the following questions should be posed during the planning phase of a performance audit.

• What is the end result(s) of the program or activity being audited? (Consider the Departmental Results Framework and the Sustainable Development Goals, among other sources, in answering this question.)

• Has the entity defined the intended end result(s) in measureable terms?

• Has the entity identified measures to track its progress toward the end result(s)? (Measures could include process measures, for example, the number of days expected to deliver a service.)

• Has the entity established targets for the above measures – the specific, planned level of a result to be achieved within an explicit timeframe (e.g. 90% of client requests will be addressed within 3 days).

• Has the entity established the means to collect data for the above measures?

• Is the entity collecting data and reporting on the above measures?

• Are targets being achieved?

• Is progress being made to achieve the identified results?

Appendix A provides information about, and examples of, results measures.

In finalizing the audit plan, lines of enquiry are selected that address significant aspects of the program or activity where the team has determined there is high residual risk. The Office also wants to provide Parliamentarians and Canadians with assurance about whether the overall program or activity is achieving intended results. Accordingly, teams need to focus some of their audit resources on answering this question.

Conducting the examination

During the examination phase, the audit team should remain cognizant of the Office’s desire to report on the achievement of expected results for Canadians, or on the issues affecting the entity’s ability to measure and report on expected results. The audit team should ensure that it has done sufficient work to support observations on the questions listed in the planning of the audit section above.

Reporting the audit

One objective of the Office is to provide Parliamentarians and other users of its reports with information about the expected results of the program or activity that was audited and the measures used to assess the entity’s success in delivering results for Canadians. The Office may also make observations and recommendations regarding the quality of information systems and results frameworks that the entity has in place to support the monitoring and reporting of results.

The Office aims to monitor identified results in the future to report on the progress entities have made toward achieving expected results.

Consultation

Over the next couple of years, as the Office establishes its approach to integrating results measures into performance audits, consultation with the Principal, Strategic Planning is strongly encouraged during the preparation of the Audit Report Submission and the Audit Logic Matrix. The objective of this consultation is to support the audit team in the identification of results measures and including them in the planning of the audit, in a manner consistent with other performance audits. The desirability of further consultations during the examination and reporting of the audit would be discussed at this time.

Appendix A

Performance measures can be used to assess both efficiency and effectiveness on both quantity and quality dimensions. Results measures are those performance measures that generally assess effectiveness on the quality dimension.

	Quantity #	Quality %
Efficiency	How much service did we deliver?	How well did we deliver it?
Effectiveness	How much change did we produce?	What quality of change did we produce?
		(Results measures)

In general, a results measure will answer the question “is anyone better off?”

Education

Is Anyone Better Off?

• % Reading scores at or above grade
• % Math scores at or above grade
• % with good attendance
• % students in extracurricular activities
• Graduation rate (% of 9th graders who graduate on time 4 years later)
• % graduates in school or jobs 1, 2, 5 years after graduation
• % placement in jobs related to training

Pesticide safety

Is Anyone Better Off?

• % of required pesticide re-evaluations completed (within required time period)
• Length of time to phase out pesticides with unacceptable human health or environmental risks
• # of pesticides conditionally registered for more than 5 years

Child welfare

Is Anyone Better Off?

• # / % of foster children in stable permanent living arrangements after 6 months in care.
• # / % of repeat abuse/neglect cases
• # / % of foster children with good school attendance
• # / % of foster children reading at grade level
• Rate of adoptions stable after 6 months

Pay system implementation

Is Anyone Better Off?

• # of public servants with outstanding pay requests
• # of outstanding pay requests
• % of bi-weekly pay cheques issued with errors

Qualities of results measures

Results measures are:

• Directly linked to the outcome they are measuring
• Trackable over time
• Clearly worded, specific, understandable
• Separate, where appropriate

Directly linked

Result	Indicator
Clients have access to services in the official language of their choice	Percentage of clients who reported having access to services in the language of their choice
GIC appointments are made on a timely basis	Percentage of GIC appointments vacant for more than 60 days
Individuals entering the country are subject to a full inspection	Percentage of incidents where officers found to not have conducted a full inspection of individuals entering the country

Trackable over time

Indicator	Actual Performance
	2015	2017
Percentage of sectors monitored that saw an increase in productivity	80% (out of 5)	40% (out of 10)
Number of boxes of archival records remaining to be opened in order to make the content known and accessible to Canadians	98,000	84,000
Average response time to non-compliant cosmetic notifications	270 days	226 days

Clearly worded, specific and understandable

• Percentage of requests answered within 10 days
• Greenhouse gas emissions (in megatonnes)
• Number of priority applications accepted for every regular application

Comparable

• Canada’s ranking among G7 countries on X
• Percentage of transactions that met industry standard
• Percentage change in greenhouse gas emissions since 2005 (as compared to other countries)