Annual Report on the Practice Reviews of the Direct Engagements Completed in the 2021–22 Fiscal Year

Table of Contents

• Introduction
• Objective
• Scope
• Rating
• Results of the Reviews and Related Conclusions
• Potential Root Causes
  • Incomplete understanding of the standards
  • Insufficient business readiness to adapt
• Overall Conclusion and Proposed Next Steps

Introduction

The Internal Audit and Evaluation Office (formerly the Practice Review and Internal Audit) team continued to help the Office of the Auditor General of Canada(OAG) to meet its obligation under the Chartered Professional Accountants of Canada’s Canadian Standard on Quality Control^[1] for the cycle of direct engagements completed^[2] on or before 31 March 2022. The team did this by conducting inspection activities to determine the extent to which engagement leaders were complying with Canadian auditing standards, OAG policies, and applicable laws and regulations when conducting their audits. The team also ensured that audit reports subject to review were supported and appropriate.

Objective

The practice review’s objective is to provide the Auditor General of Canada with assurance that:

• direct engagements comply with Canadian standards on assurance engagements for direct engagements, OAG policies, and applicable laws and regulations
• audit reports are supported and appropriate

Scope

This report summarizes the reportable observations related to the practice reviews of 6 direct engagements (4 performance audits and 2 special examinations) completed in the 2021–22 fiscal year. The review of one of the four performance audits was performed by a Canadian Council of Legislative Auditors partner under the guidance and support of the Internal Audit and Evaluation Office.

---

[1] The Chartered Professional Accountants of Canada’s Canadian Standard on Quality Control 1 was replaced by the Canadian Standard on Quality Management (CSQM) 1 and the CSQM 2 as a new approach to quality management at the firm level (with the CSQM 2 providing guidance on engagement quality reviews). Firms were required to design and implement their systems of quality management by 15 December 2022 and evaluate them within 1 year following this date. All of the audits reviewed were completed and tabled before 1 April 2022, although some of the practice reviews were completed more than a year after the audits were tabled.

[2] For direct engagements (performance audits and special examinations), “completed” means performance audit reports that were presented for tabling or special examinations that were presented to the respective Crown corporation’s governing body, such as a board of directors, during the 2021–22 fiscal year. Reports substantially completed in the 2021–22 fiscal year but tabled in the 2022–23 fiscal year will be included in the next review cycle.

Rating

Each audit file reviewed was rated as one of the following:

• Compliant. The audit file is compliant in all significant respects with Canadian standards on assurance engagements for direct engagements, OAG policies, and applicable laws and regulations. Some areas of improvement may have been noted.
• Non-compliant. The audit file does not comply with Canadian standards on assurance engagements for direct engagements, OAG policies, or applicable laws and regulations. Significant deficiencies exist, and major improvements are necessary.

After completing each review, we also concluded on whether the audit report was supported and appropriate.

Results of the Reviews and Related Conclusions

This report covers the second cycle of direct engagements that were completed remotely because of the coronavirus disease (COVID‑19) pandemic and includes performance audits of the government’s response to the pandemic.

The most common and/or significant observations that came out of this review cycle involved concerns about:

• the preparation and review of key documentation before the date of the audit report
• a clear conclusion against the audit objective in the report
• the documentation, preparation, and review of audit conclusions against the audit objective, the audit criteria, and overall
• the audit logic matrices, which were to include audit programs
• the minimum requirements for quality reviewer responsibilities
• the minimum requirements for engagement leader responsibilities

The Practice Review On-Time Report of direct engagements completed in the 2021–22 fiscal year includes all anonymized observations and a number of related files. Furthermore, each engagement leader received an individual report that detailed all of the reportable observations resulting from the practice review of the engagement leader’s audit file. A separate report was provided to the quality reviewer where one was assigned and issues were noted.

Of the 6 direct engagement files reviewed, we determined that 5 audit reports were supported and appropriate and that 1 was supported and appropriate but raised concerns, as all 3 high-risk areas we reviewed had some elements that were not supported or not appropriate. However, the reviewer concluded that the main message in the report would not have been materially different. This file was the same file that was found to be non-compliant and for which issues were noted for the quality reviewer, as explained below.

We rated 5 of the files (3 performance audits and 2 special examinations) as compliant in all significant respects with Canadian standards on assurance engagements for direct engagements, OAG policies, and applicable laws and regulations. Although they were compliant in all significant respects, some areas for improvement were noted in the performance audits reviewed and were communicated to the engagement leaders accordingly.

We rated 1 file (a performance audit) as non-compliant because of deficiencies in meeting:

• the requirement for the preparation and review of both the file documentation, including the documenting of significant judgments, and of the audit logic matrices, which were to include the audit programs
• the requirement for the documentation and review of the conclusions against the audit objectives, audit criteria, and overall before the date of the report
• the minimum requirement for an engagement leader and a quality reviewer

We want to note that the engagement leader disagreed with most of the observations raised. The disagreement was also noted in the individual report issued specifically to the engagement leader. The reviewer considered the engagement leader’s comments but deemed the final practice review report to be supported and appropriate on the basis of the information and documentation contained in the audit file.

Potential Root Causes

In preparing to implement the new standards on quality management, which did not apply to this specific set of practice reviews, we started to include preliminary work to help identify potential root causes for deficiencies identified in the previous cycle’s file that had been rated as non-compliant and continued into this year’s review cycle. The extent of the work was limited but could be of value as the OAG addresses noted areas of concern. The process of, specific procedures for, and training on identifying root causes, analyzing them, and using them to devise recommendations and plans of action will need to be strengthened and become more robust as the new standards are fully implemented.

Most of the issues noted related to the file rated as non-compliant, and as such, the root causes identified below are more highly focused on this file. However, some of the root causes for this file were also noted in other performance audit files with less significant observations. A note about this is indicated below where applicable.

As additional context for this current cycle, the audit teams faced the challenges brought on by the pandemic, including but not limited to, the following:

• working from home
• higher workload from auditees, which affected their response times
• attempting to streamline processes both from auditor and auditee standpoints while delivering in a timely manner
• navigating new processes put in very quickly
• dealing with the stress brought on by the effects of the pandemic and working on a sensitive audit subject matter

As it relates to the review at the engagement level, we found 2 main root causes, which are explored below. Additional elements for consideration when assessing root causes at the firm level are explored in the Overall Conclusion and Next Steps section.

Incomplete understanding of the standards

Based on our review, we found that there was an incomplete understanding of the standards. This was noted in 2 significant ways.

First, there was a misunderstanding of the roles and responsibilities of an engagement leader and of a quality reviewer. The file deemed non-compliant indicated a larger gap for both roles, while 2 other files indicated some more minor issues relating to the engagement leader’s roles and responsibilities. The misunderstanding of roles and responsibilities also extended to other audit team members. We noted the following:

• The engagement leader indicated that the heavy workload contributed to the issues raised.
• There were gaps in ensuring that the file was appropriately documented and reviewed in a few of the files reviewed.
• On the file deemed non-compliant, the quality reviewer provided an insufficient challenge role and gaps in documentation were not raised. The quality reviewer indicated that the heavy workload (concurrent audits) contributed to the degree and rigour of their involvement in the file. However, we did not find evidence that this was raised in a manner as to mitigate the risk.

Second, decisions to streamline existing processes while respecting auditing standards (that is, knowing where and what was appropriate to not perform) were not well understood. While there was information to document when some procedures would not be performed with the intent to streamline (the “what”), the documentation lacked specificity, was incomplete to document the impact this would have on compliance with standards, and a thorough analysis of the impact of the risks and appropriate risk mitigation (the “why” and the “so what”) was not evidenced.

Insufficient business readiness to adapt

We also found that the OAG was not sufficiently business ready to modernize and transform its processes to meet the evolving audit expectations. We noted that, despite expectations to deliver and perform audits differently, the audit processes and tools for the most part remained unchanged, and we noted gaps in knowledge on how to approach and manage the change appropriately.

Attempting to change significant ways of working during the situation that requires it, rather than before the changes are needed, will almost always increase the level of risk, especially in an organization that has not significantly changed its processes in many years.

Furthermore, requiring this to be addressed on each individual file through the judgment of various engagement leaders, rather than collectively using an office-wide perspective, will expose the organization to increased risk and higher levels of inconsistency. This is made more pronounced in a situation in which there is an incomplete understanding of standards. This was noted more significantly in the file deemed non-compliant, but it was also noted in another audit file to a lesser extent. This potential root cause manifested in the following manner:

• We noted challenges in communicating and addressing concerns raised during the audit between the auditee and the audit team, as well as between the audit team and senior management.
• There were significant changes to the report throughout the reporting phase, including towards the end of the reporting phase. Important gaps were also noted in the documentation to support significant judgments in one of the audit files (the former changes contributing to the latter gaps to a certain extent).

Overall Conclusion and Proposed Next Steps

In the previous cycle of direct engagement practice reviews, there was also 1 file (a performance audit) rated as non-compliant. Given the circumstances and environment at the time, the Internal Audit and Evaluation Office had concluded that the non-compliance was likely to be an isolated incident and that there were likely no major concerns with the system of quality management as a whole to be further examined.

However, based on the additional review of audit files subject to this review cycle and ongoing file reviews in the cycle that ended in March 2023 (performed by a different function), we are revising our previous conclusions. We now conclude that there are likely more serious concerns around the system of quality control/management as it relates to the performance audits completed under the direct engagement standards. However, thus far, no reportable issues of importance have been noted for special examinations that were also performed under the direct engagement standards by the OAG’s financial audit practice.

This cycle’s results, along with issues being noted in ongoing reviews by the OAG’s Monitoring team, strongly indicate that important deficiencies need to be further assessed. This assessment would be undertaken by the Monitoring team with the primary support of the performance audit practice, the Audit Methodology team, and the Professional Development team. This report provides an assessment at the individual engagement level. Looking at these findings, along with other factors at the firm level, will help the Monitoring team to draw an overall conclusion on the OAG’s system of quality management, as that is assessed at the firm level.

However, based on the last 2 years of practice reviews, we see compelling indications that there are issues with the system of quality management that would require comprehensive and coordinated actions. Given that the practice reviews cover only a sample of audit files, an expanded targeted review of files (risk-based and focused on areas of higher concern) may be warranted to determine the extent of the issues raised and identify systemic issues to be addressed, along with any other potential ones not identified in the limited sample of files reviewed.

Based on information obtained beyond the individual file reviews, we suggest that organizational root causes for potential deficiencies in the system of quality management could include gaps in:

• the appropriate understanding of auditing standards and office methodology
• the appropriate understanding of the importance and need of alignment throughout all phases of the audit
• the training that relates to the use of professional judgment, data analytics, sampling, and data integrity and to the need for and documentation of appropriate audit evidence
• the understanding around risk management and the culture of risk aversion
• the collaboration and meaningful relationships with entities
• the collaboration and oversight between audit teams and senior management, including clarity to reduce misunderstandings in roles, responsibilities, and accountabilities
• the early and strategic investments to ensure business readiness and agility to respond to evolving auditing needs

Some recent or upcoming changes caused by the implementation of the OAG’s system of quality management, such as the enhancement of the OAG’s quality reviewer framework and guidance in response to the CSQM 2 and other important changes deriving from the CSQM 1, should help address some of the risks and concerns raised. The transformation initiatives across the OAG, especially as they relate to audit transformation, can provide an opportunity to explore different approaches and synergies to not only improve and protect the relevance of its performance audits, but also ensure their quality and compliance with standards as a collective, rather than at the individual engagement level. It would also provide an opportunity for the OAG to critically look at its methodology and its supporting tools and processes to streamline audits so that they are more efficient while ensuring strong audit quality.

Given the significant delays in finalizing this cycle of the direct engagement practice reviews, but still trying to provide as much added value and timely information to the relevant stakeholders, the Internal Audit and Evaluation Office has been having early and ongoing conversations with the Auditor General, the Deputy Auditor General, performance audit Assistant Auditors General and the OAG’s system of quality control team to support them in their next steps to address concerns identified during the practice reviews as they were ongoing.