4043 Audit Criteria

Jul-2020

What CSAE 3001 means for developing audit criteria

CSAE 3001 requires that the audit team applies suitable audit criteria that exhibit the following characteristics:

• Relevance. Relevant criteria result in an audit report that can assist Parliament or territorial legislative assemblies in their decision-making process.

• Completeness. Criteria are complete when they do not omit relevant factors that could affect decisions of the intended users.

• Reliability. Reliable criteria allow a relatively consistent measurement or evaluation of the subject matter when used in similar circumstances by different auditors.

• Neutrality. Neutral criteria are free from bias.

• Understandability. Understandable criteria result in an audit report that can be understood by the intended users.

The suitability of criteria is context-sensitive and must be appropriate to the characteristics and activities of the audited entity and depends on the circumstances of the audit.

Audit criteria can be selected or developed in a variety of ways:

• They are often based on laws and regulations because they relate to the entity or the government as a whole; for example, sections of the Immigration and Refugee Protection Act for an audit on selecting foreign workers under Canada’s immigration program.

• They are frequently derived from central agency or entity policies, directives, guidelines, and plans; for example, Treasury Board policies and Departmental Plans (formerly known as RPP). These types of criteria are developed collectively by a group that does not follow a transparent due process.

• Criteria can be based on recognized bodies of experts that follow a transparent due process including public consultation and debate; for example, standards established by the International Organization for Standardization.

• Criteria can also be based on international commitments Canada took; for example, the United Nation’s sustainable development goals.

• In some cases, criteria are specifically designed for measuring or evaluating the subject matter in the particular circumstances of the engagement.

CSAE 3001 also requires that criteria be made available to the intended users of the audit report to help them understand how the subject matter of the audit has been measured or evaluated. The OAG always discloses the criteria used in its audit reports in the “About the Audit” section of the report, and includes the sources of the criteria.

Developing the criteria

Criteria should be developed for each area to be audited (line of audit inquiry or audit project). There can be one criterion or several. They focus, wherever possible, on the results that the program, operation, system, practice, or control is expected to achieve. When wording criteria, audit team members should:

• Express each criterion as an expectation statement vis-à-vis the entity(ies) that is derived from the source authority for the criterion.

• Express the expectation in a way that enables a conclusion to be drawn against it—either expectation met or not met (as with audit objectives).

• Divide long criteria into two or more, particularly when they have more than one major component or they have different sources; for example,

  • “We expect CRA, CIC, and HRSDC to have regularly measured their service performance to identify service quality issues.”

  • “We expect CRA, CIC, and HRSDC to have reported to Parliament and the public on their service performance.”

The assessment of the situation/condition compared to the expectations set out by the criterion results in audit findings. Taken together, the audit criteria (and associated findings) should be sufficient to allow the audit team to form a conclusion against the audit objective(s).

Criteria are a key component of the audit approach (OAG Audit 4042 Audit scope and approach). In order to assess performance expectations, audit teams develop audit questions. For further information, see OAG Audit 4044 Developing the audit strategy: audit logic matrix and OAG Audit 4045 Evidence-gathering methods.

Documenting the assessment of suitability of criteria

During the process of selecting or developing suitable criteria for the audit, audit teams should document significant professional judgements made to assess the suitability of audit criteria (OAG Audit 1143 Documenting significant matters and related significant professional judgments), such as the advice received from the appropriate internal specialist (if any); the decision made to keep or not to keep some selected criteria; the various sources of criteria used or not used; some rationale that explains why the criteria are considered suitable in the context of the audit; etc.

Sources of criteria

The audit team may refer to many different sources when selecting or developing suitable criteria, including

• Laws and regulations governing the operations of the entity;
• Government and board policies;
• Good practices of the sector;
• Decisions made by the legislature or the executive branch of government;
• Key performance indicators used by the entity or the government;
• Standards developed through research or used by professional and/or international organizations;
• Benchmarks of good performance for comparative entities;
• Planning documents, contracts and budgets from the entity;
• Criteria used in similar performance audits; and
• Consultation with subject matter experts.

The criteria and their sources are disclosed in the “About the Audit” section of the report (OAG Audit 7030 Drafting the audit report). These sources determine the amount of effort needed to ensure the suitability of the criteria. When using laws or regulations as criteria, the audit team only need to ensure that they are directly related to the audit objective. The same is true of central agency or entity policies. Although central agency and entity policies are not usually subject to public debate, they are based on consultation within government and are authoritative.

Directives, guidelines, plans, tools, controls, and measures developed by central agencies, Crown corporations, and government departments and agencies are less authoritative. However, they can be used as criteria if the audit team can validate their suitability through sufficient research and validation. The audit team can consult with professional bodies or other organizations carrying out similar activities or operations to test the quality of the standards or to identify best practices.

Criteria developed specifically for the audit include criteria based on performance data from other organizations, inside or outside the federal government, that have

• comparable activities or operations,
• best practices determined through benchmarking or consultation, and
• standards the auditors developed by analyzing a task or activity.

These types of criteria require the most effort by the team to ensure their suitability.

Over the years, the OAG has developed and tested criteria for a large number of departments, Crown corporations, agencies, programs, and operational areas. However, the fact that these criteria have been used in the past does not, by itself, make the criteria authoritative. It is the audit team’s responsibility to reassert the source and suitability of the criteria.

Frequently, a criterion is based on more than one source. Auditors should document how the criterion was derived from multiple sources.

Audited entity’s acknowledgement of the suitability of criteria

The OAG seeks entity management’s acknowledgement of the suitability of the criteria. For performance audits, criteria are presented to the entity as part of the audit plan summary (OAG Audit 4090 Audit plan summary for performance audits), and for special examinations as part of the special examination plan (OAG Audit 4100 Special examination plan). The entity is given an opportunity to comment on the criteria, and the team may make changes as a result. The deputy head (for performance audits) or the head of the Crown corporation (for special examinations) is asked to acknowledge in writing the suitability of the criteria.

If the team is unable to obtain acknowledgement from the entity’s management that the criteria are suitable, the engagement leader must assess the impact on the audit work and the audit report. A clear case must exist and be documented on why, despite objections by the entity, the engagement leader feels the criteria are suitable in the circumstances. Under no circumstances is the audit to be carried out using criteria that would result in biased or misleading audit results. If there is disagreement with management about the criteria, this should be disclosed in the audit report with an explanation of why the audit team used the criteria despite management’s objection.

As the audit progresses, additional information may result in a criterion not being necessary for achieving the audit objective. In these circumstances, further audit work related to the criteria is not needed; however, the team should document the reason for eliminating a criterion in the audit file and also notify the entity. Eliminated criteria do not appear in the audit report.

Follow-up work and criteria

Follow-up audit work examines the recommendations or significant findings made in previous OAG audits. The previous recommendations or findings serve as audit criteria. Further, commitments made by entities in response to audit recommendations may also serve as criteria. However, if there are redefined or additional issues being considered, the team needs to formulate new criteria. For further information, see OAG Audit 4042 Audit scope and approach, under the “Inclusion of follow-up work” section.