Performance Audit Manual

COPYRIGHT NOTICE — This document is intended for internal use. It cannot be distributed to or reproduced by third parties without prior written permission from the Copyright Coordinator for the Office of the Auditor General of Canada. This includes email, fax, mail and hand delivery, or use of any other method of distribution or reproduction. CPA Canada Handbook sections and excerpts are reproduced herein for your non-commercial use with the permission of The Chartered Professional Accountants of Canada (“CPA Canada”). These may not be modified, copied or distributed in any form as this would infringe CPA Canada’s copyright. Reproduced, with permission, from the CPA Canada Handbook, The Chartered Professional Accountants of Canada, Toronto, Canada.

6040 Selection of Items for Review
May-2024

Overview

The objective of the engagement teams when selecting items for review is to obtain sufficient and appropriate evidence to conclude against the audit objective.

CPA Canada Copyright

CSAE 3001 Requirements

53R. Based on the practitioner's understanding (see paragraph 51R) the practitioner shall: (Ref: Para. A110-A114)

[…]

(b) Design and perform procedures to respond to the assessed risks and to obtain reasonable assurance to support the practitioner's conclusion. In addition to any other procedures on the underlying subject matter that are appropriate in the engagement circumstances, the practitioner's procedures shall include obtaining sufficient appropriate evidence as to the operating effectiveness of relevant controls over the underlying subject matter when:

(i) The practitioner intends to rely on the operating effectiveness of those controls in determining the nature, timing and extent of other procedures, or

(ii) Procedures other than testing of controls cannot alone provide sufficient appropriate evidence.

[…]

68. The practitioner shall evaluate the sufficiency and appropriateness of the evidence obtained in the context of the engagement and, if necessary in the circumstances, attempt to obtain further evidence. The practitioner shall consider all relevant evidence, regardless of whether it appears to corroborate or to contradict the measurement or evaluation of the underlying subject matter against the applicable criteria. If the practitioner is unable to obtain necessary further evidence, the practitioner shall consider the implications for the practitioner’s conclusion in paragraph 69. (Ref: Para. A147-A153)

CSAE 3001 Application Material

A110. The practitioner chooses a combination of procedures to obtain reasonable assurance or limited assurance, as appropriate. The procedures listed below may be used, for example, for planning or performing the engagement, depending on the context in which they are applied by the practitioner:

Inspection;
Observation;
Confirmation;
Recalculation;
Reperformance;
Analytical procedures; and

A114. An assurance engagement is an iterative process, and information may come to the practitioner’s attention that differs significantly from that on which the determination of planned procedures was based. As the practitioner performs planned procedures, the evidence obtained may cause the practitioner to perform additional procedures.

Sufficiency and Appropriateness of Evidence (Ref: Para. 14(j), 68)

A147. Evidence is necessary to support the practitioner’s conclusion and assurance report. It is cumulative in nature and is primarily obtained from procedures performed during the course of the engagement. It may, however, also include information obtained from other sources such as previous engagements (provided the practitioner has determined whether changes have occurred since the previous engagement that may affect its relevance to the current engagement) or a firm's quality policies or procedures for acceptance and continuance of client relationships and assurance engagements. Evidence may come from sources inside and outside the appropriate party(ies). Also, information that may be used as evidence may have been prepared by an expert employed or engaged by the appropriate party(ies). Evidence comprises both information that supports and corroborates aspects of the underlying subject matter, and any information that contradicts aspects of the underlying subject matter. In addition, in some cases, the absence of information (for example, refusal by the appropriate party(ies) to provide a requested representation) is used by the practitioner and, therefore, also constitutes evidence. Most of the practitioner's work in forming the assurance conclusion consists of obtaining and evaluating evidence.

A148. The sufficiency and appropriateness of evidence are interrelated. Sufficiency is the measure of the quantity of evidence. The quantity of evidence needed is affected by the risks of the underlying subject matter containing a significant deviation (the higher the risks, the more evidence is likely to be required) and also by the quality of such evidence (the higher the quality, the less may be required). For certain types of direct engagements such as performance audits, there may also be a higher risk of concluding that there is a significant deviation when that is not the case. The appropriateness of the practitioner’s decision regarding whether a matter identified is a significant deviation is affected by the quantity and quality of evidence obtained.

A149. Appropriateness is the measure of the quality of evidence; that is, its relevance and its reliability in providing support for the practitioner’s conclusion. The reliability of evidence is influenced by its source and by its nature, and is dependent on the individual circumstances under which it is obtained. Generalizations about the reliability of various kinds of evidence can be made; however, such generalizations are subject to important exceptions. Even when evidence is obtained from sources external to the appropriate party(ies), circumstances may exist that could affect its reliability. For example, evidence obtained from an external source may not be reliable if the source is not knowledgeable or objective. While recognizing that exceptions may exist, the following generalizations about the reliability of evidence may be useful:

Evidence is more reliable when it is obtained from sources outside the appropriate party(ies).
Evidence that is generated internally is more reliable when the related controls are effective.
Evidence obtained directly by the practitioner (for example, observation of the application of a control) is more reliable than evidence obtained indirectly or by inference (for example, inquiry about the application of a control).
Evidence is more reliable when it exists in documentary form, whether paper, electronic, or other media (for example, a contemporaneously written record of a meeting is ordinarily more reliable than a subsequent oral representation of what was discussed).

A150. The practitioner ordinarily obtains more assurance from consistent evidence obtained from different sources or of a different nature than from items of evidence considered individually. In addition, obtaining evidence from different sources or of a different nature may indicate that an individual item of evidence is not reliable. For example, corroborating information obtained from a source independent of the appropriate party(ies) may increase the assurance the practitioner obtains from a representation from the appropriate party(ies). Conversely, when evidence obtained from one source is inconsistent with that obtained from another, the practitioner determines what additional procedures are necessary to resolve the inconsistency.

A151. In terms of obtaining sufficient appropriate evidence, it is generally more difficult to obtain assurance about the underlying subject matter covering a period than about underlying subject matter at a point in time. In addition, conclusions provided on processes ordinarily are limited to the period covered by the engagement; the practitioner provides no conclusion about whether the process will continue to function in the specified manner in the future.

A152. Whether sufficient appropriate evidence has been obtained on which to base the practitioner's conclusion is a matter of professional judgment.

OAG Policy

When representative sampling is used in a reasonable assurance engagement, at a minimum, audit samples shall be sufficient to attain a confidence interval of 10% and a confidence level of 90%. Sampling for audits of high risk or sensitivity shall be sufficient to attain the confidence interval of 5% and confidence level of 95%. [Jul-2019].

OAG Guidance

When planning and conducting a direct engagement, engagement teams are required to obtain sufficient and appropriate evidence on which to base audit conclusions.

Selection of items for review

Engagement teams can rarely examine an entire population of items, which necessitates their review of just a selection. The 2 fundamental types of selection are audit sampling and purposeful selection.

Exhibit 1—Types of selection

Selection type	Definition	Purpose
Audit sampling Also referred to as representative sampling or generalizable sampling	Audit sampling is the application of auditing procedures to a representative group of less than 100% of the items within a population of audit relevance such that all sampling units have a chance of selection in order to provide the auditor with a reasonable basis on which to draw conclusions about the entire population.	Conclude on the overall population of items.
Purposeful selection Also referred to as targeted testing	Purposeful selection is used to comment or conclude specifically on the items chosen for examination. There are 2 ways to select items: judgmental selection and random selection.	Conclude specifically on the items selected. Purposeful selection has no statistical basis for concluding on an entire population of items.

Selection type

Definition

Purpose

Audit sampling

Also referred to as representative sampling or generalizable sampling

Audit sampling is the application of auditing procedures to a representative group of less than 100% of the items within a population of audit relevance such that all sampling units have a chance of selection in order to provide the auditor with a reasonable basis on which to draw conclusions about the entire population.

Conclude on the overall population of items.

Purposeful selection

Also referred to as targeted testing

Purposeful selection is used to comment or conclude specifically on the items chosen for examination. There are 2 ways to select items: judgmental selection and random selection.

Conclude specifically on the items selected.

Purposeful selection has no statistical basis for concluding on an entire population of items.

Engagement teams need sampling or selection plans whenever an audit procedure is designed to include audit sampling or purposeful selection.

Questions to consider when building a sampling or selection plan

If your team is considering selecting items to review, see below for a list of questions your team should consider.

Exhibit 2—Questions to consider for selection plan

Consideration	Question
Purpose of the selection or sample	What audit question does each selection or sample address? What item is the team examining (for example, file, case, project)? What procedures will the team perform on each item selected? How are errors or deviations defined? How will the procedures performed contribute to concluding against the audit objective?
Population characteristics	What population is the team examining? For what period of time? What is the size of the population? How will the entity provide the population for selection to the team (Excel workbook, Access database, physical records stored on site)? What procedures will the team undertake to provide assurance on the accuracy and completeness of data provided by the entity or through other sources (for example, review of entity data quality or data assurance program or controls, comparison with control totals, gap analysis, screening for extreme or invalid values)? See the questions listed related to reliability and accuracy of data in section OAG Audit 6020 Assessing the Reliability of Data. If the team is using secondary evidence for sampling, how will it get assurance about the data integrity of the information it receives from others? Are the characteristics of the population homogeneous? In other words, do they share similar attributes being tested? If not homogeneous, what procedures will the team adopt to address this, particularly if audit sampling is used?

Consideration

Question

Purpose of the selection or sample

What audit question does each selection or sample address?
What item is the team examining (for example, file, case, project)?
What procedures will the team perform on each item selected?
How are errors or deviations defined?
How will the procedures performed contribute to concluding against the audit objective?

Population characteristics

What population is the team examining? For what period of time?
What is the size of the population?
How will the entity provide the population for selection to the team (Excel workbook, Access database, physical records stored on site)?
What procedures will the team undertake to provide assurance on the accuracy and completeness of data provided by the entity or through other sources (for example, review of entity data quality or data assurance program or controls, comparison with control totals, gap analysis, screening for extreme or invalid values)? See the questions listed related to reliability and accuracy of data in section OAG Audit 6020 Assessing the Reliability of Data.
If the team is using secondary evidence for sampling, how will it get assurance about the data integrity of the information it receives from others?
Are the characteristics of the population homogeneous? In other words, do they share similar attributes being tested?
If not homogeneous, what procedures will the team adopt to address this, particularly if audit sampling is used?

Framework for selecting items for review

The following chart provides a framework describing the types of item selection.

Exhibit 3—Framework for audit sampling and purposeful selection

Audit sampling

Audit sampling should be considered in the following circumstances:

There is a well-defined population that is relevant to the audit.
It is impractical to examine the entire population.
The population is accessible.
The representation of that population is reliable (that is, accurate and complete).
The team has the audit resources it needs.

Audit sampling is very efficient with medium and large populations, but it can be inefficient for small populations. If teams are considering audit sampling with populations of less than 100, they should consult with the OAG Internal Specialist, Research and Quantitative Analysis.

Confidence interval and confidence level

The OAG’s policy is that all audit sampling should be performed with a confidence level of 90% or above and a confidence interval of 10% or below based on the assessed risk of significant deviation from the applicable criteria used to measure or evaluate the audit objectives. The engagement team shall consider higher confidence levels and more precise confidence intervals when the risk of significant deviation from the applicable criteria is assessed to be high.

Two measures inform about how effective a sample is likely to be as an estimator of a population: confidence interval and confidence level.

The confidence interval (also known as the margin of error) is the range of values with which the engagement team can be reasonably confident that the true population value resides, given the result from the sample. Smaller, or narrower, confidence intervals mean that there is greater precision around the estimate of a population. For example, the narrower confidence interval of +/-5 is more precise than the wider confidence interval of +/-10. In the first case, the engagement team may be confident that the true population value lies from 5 below the sample value to 5 above it. In the second case, the team would be confident only that the population value rests within the interval from 10 below the sample value to 10 above it, a much wider range. This wider range means that the confidence interval is less precise.

The confidence level is a statistical measure of the confidence that the true population value is within the specified confidence interval. The confidence level does not relate to the sample value. It relates to the confidence interval for the sample value.

Confidence level and confidence interval are closely interrelated, and both help to determine what the sample size should be. We typically compute our sample sizes so that we achieve particular targets for confidence interval and confidence level. Per our policy, when audit sampling is used, at a minimum, audit samples shall be sufficient to attain a confidence interval of 10% and a confidence level of 90%. Sampling for audits of high risk or sensitivity shall be sufficient to attain the confidence interval of 5% and confidence level of 95%.

So, given a sample value of 30% non-compliance, we might say that we are 90% confident that the true population value is between 10 less (20%) and 10 more (40%) using the OAG’s minimum requirements.

Expected error and tolerable error

Expected error is the best estimate of the error or deviation that exists in a population. Tolerable error is the criterion against which we are assessing the error or deviation we find. For example, if an entity has a performance standard that no more than 20% of transactions take longer than a target of 5 days, then the tolerable error would be 20 Teams might then look at prior years as a guide to expected error. For example, if the average percentage of transactions exceeding 5 days for the past 3 years was 5%, then teams might take 5% as an estimate of expected error. If, however, volume had sharply increased or processing had become more complex, teams might increase the expected error to account for these changes. As the expected error increases to approach the tolerable error, sample sizes can become very large. However, if teams underestimate expected error, they will need to go back and extend their sample, as the sample is then not sufficient in size to assess the population.

Types of audit sampling

There are 2 types of audit sampling: attribute sampling and variables sampling. Attribute sampling is the sampling approach most commonly used by direct engagement teams; variables sampling is more commonly used for financial audits.

Exhibit 4—Considerations for each type of sampling

Consideration	Attribute sampling	Variables sampling
Purpose	Assess the proportion of an attribute of interest to the auditor.	Extrapolate amounts or quantities.
Use	Can be used for both direct engagements and financial audit.	Most often used for financial audit.
Example	An attribute might be whether transactions are in error or not in error, whether individuals are male or female, whether offices are open or closed, or whether a control procedure was done properly or not.	The amount of error in salary among employees of the OAG.
Suitability	Not suitable for the extrapolation of amounts, such as dollars in error.	Not suitable for the extrapolation of the proportion of items in error.
Information needed	Confidence interval Confidence level Population size Expected proportions for the attributes tested (for example, 50% error and 50% non-error) If assessing against a criterion, the following information is also needed: Tolerable error	Confidence interval Confidence level Population size Estimated standard deviation of the variable being assessed or estimated If assessing against a criterion, the following information is also needed: Tolerable error
Sample size	Note that sample sizes are the largest when the attribute split is roughly 50%-50%.
Common methods to assess against a criterion value	Controls testing Accept-reject testing Regularly used for special examinations and financial audits. Can be used for direct engagements. Normally, controls testing and accept-reject testing are used when the expected error is 0 or very low. See the OAG Audit 6050 in the Annual Audit Manual for more information on controls testing and 7043 for accept-reject testing.	Dollar unit sampling Non-statistical sampling Regularly used in financial audits. Can increase the efficiency of variables sampling. Can be used for direct engagements. See OAG Audit 7044.2 in the Annual Audit Manual for more information on dollar unit sampling and OAG Audit 7044.1 for non-statistical sampling.

Sample size for items to be selected for review

Two common methods of attribute sampling are presented in the above table: Controls testing and accept-reject testing. References to the annual audit manuals are relevant except for the tables to help engagement teams identify the sample sizes for direct engagement.

Controls testing

Direct engagement sample sizes for controls testing slightly differ from those used in OAG Audit 6053 in the Annual Audit Manual. In order to obtain a high level of assurance, the sample sizes when testing a control are presented in the table below.

Exhibit 5—Sample sizes for high levels of assurance

Frequency of control	Assumed population of controls	Number of items to test for a high level of assurance at 90% confidence level	Number of items to test for a high level of assurance at 95% confidence level
Annual	1	1	1
Quarterly	4	2	2
Monthly	12	2	5
Weekly	52	5	15
Daily	250	20	40
Multiple times per day	More than 250	25	60

Accept-reject testing

Direct engagement sample sizes for accept-reject testing differ greatly from those used in OAG Audit 7043.1 and are presented below.

Exhibit 6—Number of items to test for high levels of assurance

Desired level of assurance	Number of items to test for 0 exceptions tolerated	Number of items to test for 1 exception tolerated	Number of items to test for 2 exceptions tolerated
High level of assurance (90%, 95%)	25, 60	40, 95	55, 135

Engagement teams can consult the OAG Internal Specialist, Research and Quantitative Analysis, to confirm the sampling approach when needed.

The Annual Audit Manual also includes useful information in section OAG Audit 6052 Nature of controls tests and in section OAG Audit 7043.1 A five step approach to performing accept-reject testing. While not directly applicable to direct engagements, engagement teams can consult those sections to deepen their knowledge and understanding of the nature of controls tests and the accept-reject testing.

Application of sampling methods

Both attribute and variables sampling can be applied in 2 ways: to estimate a population value or to assess against a criterion.

Exhibit 7—Descriptions and examples of application

Consideration	Estimate a population value	Assess against a criterion
Purpose	Intended to project the extent of error or deviation observed in the sample as an estimate of the population error or deviation. For example: What is the average time to completion for construction projects commissioned by Defence Construction Canada (variables sampling)? What proportion of construction projects were completed within their target (attribute sampling)?	Intended to assess whether errors or deviations in the population exceeded a specified criterion value. For example: Did Immigration Refugees and Citizenship Canada meet its performance standard of no more than 20% of citizenship applications exceeding published performance standards (attribute sampling)? Was the total misstatement in salary expenses at the OAG within 100,000 (variables sampling)? It may not be appropriate to project the extent of error or deviation observed in the sample as an estimate of the population error or deviation. Teams should consult the data analytics and research methods team to learn about this option.
Example	Estimating the proportion of employees with errors in salary payments (attribute sampling) or the amount of error in salary payments (variables sampling).	Assessing whether the proportion of employees with errors in salary exceeds 10% (attribute sampling) or whether the amount of error in salary payment exceeds $500 million (variables sampling).
Use	Used for direct engagements and financial audits.	Sampling to assess against a criterion may provide efficiencies in performance audit. This sampling is most often used for special examinations and financial audits

Consideration

Estimate a population value

Assess against a criterion

Purpose

Intended to project the extent of error or deviation observed in the sample as an estimate of the population error or deviation. For example:

What is the average time to completion for construction projects commissioned by Defence Construction Canada (variables sampling)?
What proportion of construction projects were completed within their target (attribute sampling)?

Intended to assess whether errors or deviations in the population exceeded a specified criterion value. For example:

Did Immigration Refugees and Citizenship Canada meet its performance standard of no more than 20% of citizenship applications exceeding published performance standards (attribute sampling)?
Was the total misstatement in salary expenses at the OAG within 100,000 (variables sampling)?

It may not be appropriate to project the extent of error or deviation observed in the sample as an estimate of the population error or deviation. Teams should consult the data analytics and research methods team to learn about this option.

Example

Estimating the proportion of employees with errors in salary payments (attribute sampling) or the amount of error in salary payments (variables sampling).

Assessing whether the proportion of employees with errors in salary exceeds 10% (attribute sampling) or whether the amount of error in salary payment exceeds $500 million (variables sampling).

Use

Used for direct engagements and financial audits.

Sampling to assess against a criterion may provide efficiencies in performance audit.

This sampling is most often used for special examinations and financial audits

Purposeful selection

Purposeful selection is used to comment or conclude specifically on the items chosen for examination. Engagement teams performing direct engagements often encounter situations that are more amenable to purposeful selection, not audit sampling. This may be because significant portions of those populations are inaccessible to the audit, or the populations consist of many distinct smaller sub-populations. Also, the systems and processes and the controls that are applied to items in these 2 groups may be entirely different.

Exhibit 8—Types of purposeful selection

There are 2 types of purposeful selection: judgmental sampling and random sampling. Direct engagement teams often use a combined approach.

Judgmental sampling relies on specific criteria that the team uses to select individual items for examination or to scope particular types of items for possible selection.
Random selection is used when we do not have a particular approach in mind. Often, judgmental selection is used to define a particular set for potential examination, and then random selection is used to choose the items for examination.

A combined approach might alternate selections between a random item and the remaining highest-value item until all items have been selected.

Note: Engagement teams cannot extrapolate their findings to the overall population. Often, items are selected and examined as case studies, allowing auditors to focus on the particular aspects of each item that made it noteworthy for selection.

Documenting your sampling plan

Engagement teams document their detailed sampling plan including

a statement about what the evidence will allow the engagement team to report
the desired accuracy of the results
the population size
the size of the set of items or sample
the selection or sampling criteria

Breadcrumb

Performance Audit Manual

6040 Selection of Items for Review
May-2024

Overview

CSAE 3001 Requirements

CSAE 3001 Application Material

OAG Policy

OAG Guidance

Selection of items for review

Audit sampling

Types of audit sampling

Sample size for items to be selected for review

Purposeful selection

Documenting your sampling plan

Primary navigation (left column)

Institutional links

Direct Engagement Manual

Footer

Breadcrumb

Performance Audit Manual

6040 Selection of Items for Review May-2024

Overview

CSAE 3001 Requirements

CSAE 3001 Application Material

OAG Policy

OAG Guidance

Selection of items for review

Audit sampling

Types of audit sampling

Sample size for items to be selected for review

Purposeful selection

Documenting your sampling plan

Related Sections

Primary navigation (left column)

Footer

6040 Selection of Items for Review
May-2024