Performance Audit Manual
COPYRIGHT NOTICE — This document is intended for internal use. It cannot be distributed to or reproduced by third parties without prior written permission from the Copyright Coordinator for the Office of the Auditor General of Canada. This includes email, fax, mail and hand delivery, or use of any other method of distribution or reproduction. CPA Canada Handbook sections and excerpts are reproduced herein for your non-commercial use with the permission of The Chartered Professional Accountants of Canada (“CPA Canada”). These may not be modified, copied or distributed in any form as this would infringe CPA Canada’s copyright. Reproduced, with permission, from the CPA Canada Handbook, The Chartered Professional Accountants of Canada, Toronto, Canada.
1510 Selection of Performance Audit Topics
Dec-2023
Overview
The selection of performance audit topics is a strategic decision that contributes to the mission and vision of the Office of the Auditor General of Canada (OAG). It is an iterative and incremental process to select topics that are relevant and timely for parliamentarians and Canadians. Currently, the audit selection process is going through a multistage transformation.
OAG Guidance
The Auditor General may bring to the attention of Parliament or a territorial legislative assembly those matters that the Auditor General believes to be of interest and significance. This can be done through performance audit reports or other products, including but not limited to reviews, studies, or follow‑ups on past recommendations or on the government’s performance results in areas we previously audited. The starting point in selecting products that can add value is deciding what to audit from among the multitude of government activities and programs. The OAG can examine activities or programs within one entity, across several entities, or across all of government and may look at the role of central agencies in regard to the activities or programs to be examined. To assess the state of a program in Canada, the OAG can also undertake audits in cooperation or collaboration with other levels of government, such as provincial auditor general offices. Audits can also examine the implementation of findings and recommendations from previous audits.
Performance audit evergreen selection process
The objective of the performance audit selection process is to select audits that are relevant and timely to parliamentarians, members of legislative assemblies, and Canadians. The evergreen audit selection process is intended to be an agile process that can respond to changes quickly and ensure that audit topics that are selected add value for the OAG’s stakeholders.
The OAG is building an inventory of audit proposals to select from as an evergreen process. This inventory of audit proposals is ranked according to criteria (see below) to facilitate approval of audits. The process includes the following steps:
- Environmental scan—An environmental scan, which incorporates the views of parliamentarians, members of legislative assemblies, Canadians, and federal and territorial entities, informs the development of the focus areas for audit proposals.
- Collection and screening of audit proposals—OAG staff submit audit proposal forms through an online portal. These proposals are screened for completeness and grouped into themes before being sent to reviewers.
- Cross-functional review of audit proposals—Groups of reviewers from across the OAG individually read and rank proposals, then meet as a team to assign a final score to each proposal.
- Workshop—Members of cross-functional review teams and senior management meet for an in-person workshop to finalize a ranked inventory of audit proposals.
- Approvals—Ranked inventory of audits proceeds through an approval process.
Step 1. Environmental scan
An environmental scan is done at the beginning of the audit selection process. The environmental scan identifies domestic and global thematic trends that are expected to affect and government in the future. With an environmental scan, the OAG identifies key focus areas that could anticipate future areas of interest and value to Parliament and territorial legislatures in the years to come. These observations point to the uncertainties and implications of various trends over the near term and the longer term. The trends do not predict the future; rather, these focus areas will help the OAG understand possible implications of these trends and direct audit selection to address them.
The 10 focus areas are as follows:
- improving Indigenous and northern well-being
- delivering on climate change accountability
- enabling financial and economic stability
- ensuring effective and secure information management and information technology and infrastructure
- implementing sustainable development and protecting the environment and human health
- modernizing and transforming public service delivery
- protecting biodiversity
- protecting Canadians
- preparing for and responding to emergencies
- supporting marginalized groups for an inclusive society
Step 2. Collection and screening of audit proposals
To obtain a broad and diverse list of potential performance audit topics, all OAG staff are invited to submit audit proposals. An online proposal form is available to facilitate the submission of topics.
The proposals go through an initial screening and are then grouped by subject matter for the cross-functional review teams.
Step 3. Cross-functional review of audit proposals
Proposals grouped by focus areas are assigned to cross-functional review teams. These review teams are groups of 6‑8 people composed of volunteers from across the OAG. Each team is chaired by an assistant auditor general.
Each review team meets to discuss the proposals and scores them using 4 criteria: relevance, significance, value for money, and outcomes (Exhibit 1).
Teams are asked to score each criterion with a value of 1, 3, or 5 and come to an agreement on a final score. A scoring grid assists reviewers in determining scores. If the review team concludes that there is a critical risk within the audit proposal that would prevent the audit from being completed, the risk is noted and the proposal receives a score of zero.
Exhibit 1—Evaluation criteria for audit proposals
Criteria | Questions to consider | What high-scoring proposals should show |
---|---|---|
Relevance |
|
|
Significance |
|
|
Value for money |
|
|
Outcomes |
|
|
Step 4. Workshop
A workshop is held to finalize a ranked inventory of audit proposals, considering additional dimensions of timing and balance (Exhibit 2) across audit proposals with the highest ranked scores. Workshop attendees include members of all cross‑functional review teams and other senior members of the OAG.
Workshop participants examine higher-scored proposals and consider timing and balance through discussion and anonymous voting. The workshop does not re-score proposals but instead, it adds additional dimensions to the score to complete the ranked order of audit proposals.
Exhibit 2—Criteria considered
Criteria | Questions to consider |
---|---|
Timing |
|
Balance |
|
Step 5. Approvals
The ranked inventory is presented to the Auditor General, the Deputy Auditor General, and the Commissioner of the Environment and Sustainable Development for discussion and approval with the participation of the assistant auditors general in the performance audit practice. They consider many dimensions, including commitments, priorities, and parliamentary requests. From that list and other considerations, an audit plan is developed—a pre-condition for continuing with the audit (OAG Audit 3011 Acceptance and continuance).
The ranked inventory of audit proposals is maintained and updated as part of the audit selection process on an ongoing basis for the development of future audit plans.
Audit submission forms
Audit submission forms are used to capture audit topics. They can be used by any employee of the OAG but are usually filled out by the teams as part of their strategic audit plans. The forms contain the following information:
-
what type of audit it will be (federal, territorial, study, other)
-
which entities will be included in the proposed audit
-
what the audit will examine
-
why the audit is important to legislators and Canadians
-
which of the United Nations’ Sustainable Development Goals and associated targets and indicators will be covered by the audit. Consult the following link: UN—Sustainable Development Goals (see the PDF or EXCEL version of the Global Indicator Framework at the bottom of the page)
-
what gender, equity, diversity, and inclusion considerations will be covered by the audit
-
whether the audited program provides services and, if so, how the audit will examine the impact on Canadians
Audits that are special requests from the Auditor General can be discussed and approved outside of the annual planning process (OAG Audit 3011 Acceptance and Continuance).
Strategic audit planning: Selection of the audit topics and sectors (Under Review)
Audit selection begins by understanding risks—both internal and external—facing government departments and agencies and the government as a whole. The engagement team may select audit topics that are important to the achievement of organizational or governmental goals or audit topics for which the risk is assessed to be high. Areas that are important but where the risk is assessed to be low might also be selected because even if the risk is low, any deviation could be important to the entity or to its stakeholders.
Knowledge of business—The engagement team must establish and maintain a good understanding of the area or entity objectives, expected results, and accountabilities. The team does some of this on an ongoing basis through its knowledge of business activities, which include meeting periodically with entity officials, reviewing key entity documents, and monitoring parliamentary and legislative committee activity and media reports (see OAG Audit 1505 Acquiring and maintaining knowledge of business for performance audits).
Strategic audit planning—A more intensive exercise to obtain critical, up‑to‑date entity information, known as the strategic audit planning process, takes place for each entity or functional area or for the whole government or territory. The strategic audit planning process promotes a consistent yet flexible approach to multi‑year audit planning that is documented and risk based. It demonstrates that the OAG is fulfilling its responsibilities and exercising independence and objectivity in selecting matters to audit. It ensures that OAG resources are focused on the areas of highest risk by considering each area’s significance and relevance to Parliament or the respective legislature. It also ensures that the audits add value to Parliament or the legislature and Canadians and that the OAG is exercising due diligence in applying the discretion provided in the Auditor General Act for selecting matters for audit. The strategic audit plans may also help the team identify other types of products that could help support accountability, transparency, and improvement of government operations.
This guidance provides suggested steps to follow to identify potential areas to audit. Teams should use their discretion and professional judgment to determine the steps they need to follow to create a meaningful strategic audit plan. The following steps can be considered as part of a tool kit to plan, conduct, and report strategic audit plans (Exhibit 3).
Overview of the strategic audit planning process
Exhibit 3—Key steps of the strategic planning process
- Identify the structure of the strategic audit plan, and determine the anticipated level of effort (hours and resources) required to complete the work.
- Notify the entity that strategic audit planning work will commence.
- Gather information, consult, and assess past work.
- Conduct document review.
- Meet with the entities’ internal audit officials.
- Interview other entity officials, subject matter experts, and stakeholders.
- Consider risks and controls.
- Convene an expert advisory committee meeting, if appropriate.
- Consolidate results.
- Communicate the results of the strategic audit planning work.
Strategic audit planning is the risk-based assessment that the OAG uses to select topics for future audit work. A risk‑based assessment is a process to identify, assess, and prioritize risks to be able to identify areas to audit. The team may assess the importance of, and risks associated with, the entity’s or area’s activities by considering factors such as the following:
-
Economic, social, and environmental impact—Programs, activities, or processes that affect a large segment of the population or vulnerable populations or that affect environmental sustainability may be considered to be more important to the entity.
-
Relevance to stakeholders—The interest shown by the legislature or other governing bodies, by management of the entity, or by the public may indicate the importance of the activity to stakeholders.
-
Diversity, consistency, and clarity of the entity’s objectives and goals—Diverse or inconsistent objectives increase the risk that the entity’s activities or programs are not operating with due regard for one or more of the principles of economy, efficiency, and effectiveness or for the environment and sustainable development. Entity objectives and goals that are not clearly defined may increase the risk that they will not be achieved because they are not understood by employees.
-
Complexity of operations—An increase in the complexity of an entity’s operations through increased variety and type of programs, functions, and activities may increase the risk that the entity does not achieve its objectives and goals or that they are not achieved efficiently, economically, or with due regard to the environment or sustainable development.
-
Complexity and quality of management information and control systems—Complex systems may be more difficult to develop, enhance, and maintain. When adequate management information systems are not maintained, proper control may not be exercised.
-
Impact of environmental or organizational change—Changes in an entity’s environment or organization can affect the continuity of operations and the understanding of priorities and processes by employees. This may increase the risk that the entity’s goals and objectives will not be achieved. Environmental changes include new government priorities, significant budget amendments, and changes to enabling legislation. Organizational changes include changes in leadership, reorganization, new initiatives, and staff turnover.
-
Financial magnitude and nature of transactions—Large dollar amounts, high transaction volumes, and transaction complexity and flow may create increased risks to the entity.
-
Management response to previously identified deficiencies—Areas where management has not made adequate improvements to address important issues raised in prior performance audits or other studies may be more important and higher risk.
-
Organizational structure—Centralization and decentralization of key activities such as budgeting, payroll, disbursements, human resource management and facility management each create their own operational risks. Similarly, program delivery through agents carries different risks than those associated with direct program delivery.
-
Program delivery method—Programs in the public sector may be delivered by policy instruments such as expenditure, regulation, and revenue-raising, may provide goods or services directly or redistribute income, and may be delivered directly or by using agents. The amount of associated risk may vary depending on the delivery method.
The objective of the strategic audit planning process is to help identify areas of significance or of a nature that would require them to be brought to the attention of the House of Commons. The process allows the teams to efficiently and effectively identify potential audit topics that
-
add value, meaning that they focus on important gaps, outcomes, or areas for improvement in federal or territorial programs
-
address federal or territorial programs’ effects on individual Canadians and other stakeholders
-
can be audited, meaning that
- the subject matter is capable of consistent measurement or evaluation against the applicable criteria
- suitable criteria exist
- the evidence needed to support the conclusion can be obtained
- the work conducted will lead to recommendations that are reasonable, are able to be implemented, and can be directed to an entity that has the responsibility and authority to act on them
Key steps for strategic audit planning
Step 1. Structure the strategic audit plan, and determine the level of effort required
The strategic audit planning process has 3 phases—planning, conducting the work, and reporting. The output from this process is a list of potential topics to audit or review. A strategic audit plan can be developed for an entity within a given portfolio or can be based on a sectoral topic. It is to be reviewed and updated periodically as needed, ideally every 3 years. To remain current and relevant, teams will be asked to review annually the list of proposed audit topics that came out of the strategic audit planning and consider emerging risks, new announcements made by government, changes to government programs, and so on. A strategic audit plan can be developed for an entity within a given portfolio or can be based on a sectoral topic. It is to be reviewed and updated periodically as needed, ideally every 3 years. To remain current and relevant, teams will be asked to review annually the list of proposed audit topics that came out of the strategic audit planning and consider emerging risks, new announcements made by government, changes to government programs, and so on. This ensures that the audits that are selected as part of the annual selection process are risk based and relevant.
If teams are updating a strategic audit plan that was previously developed, they should use their professional judgment regarding the level of effort required. All of the following steps might not be necessary if the team is requesting only a few meetings to discuss recent changes or updates to documentation. Depending on the anticipated level of effort, the strategic audit planning team and the principal, who maintains a budget for strategic audit planning work, will determine the budget (hours and resources) for completing the strategic audit plan.
There are 3 types of strategic audit plans:
-
Entity-specific—A strategic audit plan is developed for a single entity, and different areas (programs, processes, and functions) within the entity are identified for audit.
-
Sectoral—A strategic audit plan is developed around a subject and includes more than one entity, as different entities share responsibility for the subject, with each entity fulfilling a unique role. Examples of sectoral subjects include information technology, innovation, environment, and human resources.
-
Whole of government(in the case of territories)—Teams responsible for a territorial portfolio will most likely perform a strategic audit plan on activities and programs within the whole of government.
The strategic audit planning team should also determine the tool to use to document the information they gather as they develop the strategic audit plan: audit working paper software or PROxI. While both options are available to teams, using PROxI is strongly encouraged to allow for ease of access, information sharing, and the communication of the results of the strategic audit plan.
Team members are not required to complete and sign an independence form, as the strategic audit planning process is not an assurance engagement performed in accordance with professional standards.
Step 2. Send notification letter
Strategic audit planning teams are required to send a notification letter to each entity included in strategic audit planning work. The letter advises the entity that strategic audit planning work will commence to assist the OAG with its selection of future audit topics. The letter also advises on solicitor-client privilege matters and on the OAG’s right to access the information needed to conduct its strategic audit planning work.
Step 3. Gather information, consult, and assess past work
Strategic audit planning work involves several steps. Teams are encouraged to share information with other strategic audit planning teams to improve their understanding and reduce duplication of work.
Identify previous strategic audit plans on the subject matter. Review the work done in any previous strategic audit plans. The information gathered will assist the team with updating an existing strategic audit plan or developing a new strategic audit plan and will ensure that the team has a baseline understanding before meeting with entity officials. The same type of work is required whether a strategic audit plan exists or not. However, if one exists, the same level of effort may not be required, as the team has a base to start from. A list of the strategic audit plans planned and completed is posted on the INTRAnet (see list of completed and scheduled strategic audit plans).
Discuss past audit work with performance audit teams. Discussions with performance audit teams may help the strategic audit planning team better understand what work has been conducted and identify whether significant risks from previous audits should be followed up. Teams can further discuss with auditors who have experience on the portfolio or have relevant expertise. Relevant documentation should also be identified and shared between the performance audit team and the strategic audit planning team, especially if the topic is a continuation of a former audit.
Discuss current audit work with financial audit teams assigned to the entity. Although collaboration with financial audit teams occurs throughout the year, specific discussions and collaboration with financial audit teams may assist the strategic audit planning team with identifying key risks.
Discuss audit work with special examination teams. Teams should also consider speaking with audit teams responsible for special examinations. Discussions with special examination teams are particularly relevant if a Crown corporation has shared responsibility with an entity on a particular subject.
Discuss risks with internal specialists. Teams should determine which internal specialists could assist in identifying specific entity or sectoral risks that should be considered. Similarly, strategic audit planning teams should share with the relevant internal specialists the risks identified through the strategic audit planning process that are related to the specialist’s area and discuss the assessment of these risks.
Teams should complete the Environment and Sustainable Development Risk Profile for Strategic Audit Planning form and consult with the Internal Specialist—Environment and Sustainable Development. Teams should also consult the audit guidance on the United Nations’ Sustainable Development Goals (SDGs) to explore links between SDGs and the entities’ mandate, core activities, and major programs.
Teams should consult the Internal Specialist—Environment and Sustainable Development to discuss any considerations of gender-based analysis plus (GBA Plus) in the strategic audit planning work.
As needed, teams should involve the data analytics team early to ensure the strategic audit planning team is aware of the questions to ask the entity to assist with understanding the quality of the entity’s data and understanding if and how the data could be used in future audits.
Teams should also involve Legal Services early in the strategic audit planning process. Legal Services can help the team navigate certain constitutional, jurisdictional, or other legal mandate issues before the planning process advances too far.
Consult with other portfolio teams. Consultations and collaboration with other teams can help determine if there are any common or shared topics or risks that may affect the development of the strategic audit plan. Strategic audit planning teams should collaborate, coordinate, and share information to efficiently and effectively identify potential audit topics. Strategic audit planning teams are also encouraged to collaborate when requesting interviews with entity officials.
External collaborations. Teams should consider external collaborations while planning the strategic audit planning work. Examples of such collaborations include consultations with the provincial legislative audit offices or other supreme audit institutions.
Other services. Teams are encouraged to consult other services available at the OAG to help them in their understanding of the government programs and activities. The OAG Knowledge Centre Library can set up media alerts, analyze results, and help conduct research on various topics. Other services, such as the Information Technology Audit team, the Controls Assurance team, Contracting and Procurement, Parliamentary Liaison, and many more, can help teams gain knowledge, skills, and experience on various topics.
Step 4. Conduct document review
The strategic audit planning team should obtain sufficient knowledge of the entity or sector so that it can identify the mandate, key activities, and key outcomes and assess any risks (Exhibit 5).
Exhibit 5—Areas for the strategic audit planning team to consider to increase its understanding of the entity or sector
Areas to develop an understanding of | Areas to consider | Examples of documentation to review or information to obtain |
---|---|---|
Operating environment |
|
|
Organizational structure |
|
|
Parliamentary interest and media attention |
|
|
The strategic audit planning team may also find it helpful to consult key experts and stakeholders who have knowledge of the entity or sector to use their expertise to confirm identified risks.
Step 5. Meet with the entities’ internal audit officials
The strategic audit planning team arranges an introductory meeting with the entity’s chief audit executive or equivalent to discuss the strategic audit planning process. The chief audit executive is responsible for the internal audit function, which acts as a liaison with entity officials. The strategic audit planning team can use this opportunity to
- explain the importance of the strategic audit planning work to plan future audits
- discuss the list of entity officials to interview
- request relevant documentation not publicly available
- decide on a working protocol
At meetings with officials of the internal audit function, the strategic audit planning team should also discuss, among other things,
- the activities performed, or to be performed, by the internal audit function
- the knowledge of business work performed by the internal audit function
- the risk and control assessments done
- evidence of non-compliance with relevant ethical requirements or fraud, if any
- other activities or information that could be shared with the strategic audit planning team
Step 6. Interview other entity officials, subject matter experts, and stakeholders
The strategic audit planning team interviews selected entity officials for strategic insight and to provide an opportunity to engage them in the ongoing strategic audit planning work. Other key individuals may be interviewed to gain a comprehensive understanding of the entity and its operations. Consideration should be given to meeting with the following individuals:
-
the deputy minister and assistant deputy ministers
-
departmental audit committee members (private meeting)
-
selected members of senior management
-
operational and program managers
-
key regional officials
-
experts and stakeholders familiar with the entity and its operations (such as non-governmental organizations, associations, businesses, and professional bodies)
-
Treasury Board officials, especially when multiple entities are involved
-
research staff from relevant House of Commons and Senate committees (the strategic audit planning team should consult with OAG Parliamentary Liaison staff to help with scheduling meetings with research staff)
Step 7. Consider risks
The strategic audit planning team should assess the risks of the entity not achieving its objectives. The higher the risk, the more likely the entity’s operations will be significantly affected.
Step 8. Convene an expert advisory committee meeting, if appropriate
The strategic audit planning team may wish to convene an expert advisory committee meeting to discuss the information obtained and the potential audit topics, including the rationale for identifying each topic. If a meeting is convened, the Auditor General, the Commissioner of the Environment and Sustainable Development, and the assistant auditors general should be included in the discussion to gain a better appreciation of the risks and the importance of the topics identified.
Step 9. Consolidate results
Once the detailed work has been completed, the strategic audit planning team summarizes the results and creates a list of potential audits or other products. The list of potential topics should include products to be conducted over subsequent tabling years.
In addition to the potential areas to examine, the following information could also be included to support the list:
- entity mandate, priorities, objectives, programs, and services provided
- key risks, challenges, significant changes, or gaps in audit coverage
- stakeholders’ concerns and interests
- importance of auditing the topic (include examples of what is not working well and the impact)
- scope of the audit (what the audit will examine and what will be excluded)
- the audit’s impact on how services are delivered (consider timeliness, accessibility, and consistency)
- applicable SDGs, targets, and indicators to be covered by the audit
- potential key messages
- value added (assurance, advice, information, and other benefits)
- potential collaboration possible with provincial or international audit offices
- possibility of contracting an entire external audit team to perform the audit
Follow-up audit work—A strategic audit plan may also cover previously audited topics. Issues are examined again if they continue to be of interest to Parliament or pose a significant risk. The team ensures that enough time has elapsed since the original audit to allow the entity to implement recommendations or address issues. Other related issues that have emerged since the previous audit may also be included if they are significant and of interest to Parliament.
Products—The team should consider the best way to add value in the areas of significance. Consideration should be given to size, scope, and type of products, such as audits, studies, reviews, and updates on performance results in areas we previously audited.
Format—The final strategic audit plan could take a variety of formats, including, but not limited to
- a PowerPoint presentation
- a working paper summarizing the work undertaken and recommendations for future audits
- a list of audit topics (using the Audit Report Submission form)
The completed plan is presented to the Auditor General, the Commissioner of the Environment and Sustainable Development, and the assistant auditors general for review and discussion.
Step 10. Communicate the results of the strategic audit planning work
Once the strategic audit plan has been finalized, it should be presented to the entity’s senior management or departmental audit committee.
Equally importantly, this information should be shared with the performance audit practice and the financial audit practice.