6054 Testing automated controls

Sep-2022

OAG Guidance

Applying a risk-based approach, develop an appropriate strategy for testing automated information processing controls, IT General Controls, and manual controls, since there are a number of interdependencies. As discussed in OAG Audit 5035.2, the ability to rely on the proper and consistent operation of automated controls usually depends on the effective operation of related ITGCs. See OAG Audit 5035.2 for further information on the linkage of ITGCs to automated controls and how ITGCs contribute evidence to the audit. There are a number of factors to consider before we determine the nature, timing and extent of testing for automated information processing controls and ITGCs, as follows:

• The quality and effectiveness of the IT control environment and ELCs over IT.

• Knowledge gained from past audits and any significant known or anticipated changes to people, processes, applications, technologies, operations or business conditions that could impact our audit.

• High-level controls executed by IT management in the normal course of business to monitor controls.

• The risks associated with the automated information processing controls and the ITGCs. The risks associated with automated information processing controls include: (a) the inherent risk of material misstatement in the underlying accounts, and (b) risk that the automated information processing controls will fail to prevent or detect material misstatement in the accounts after considering the effectiveness of other controls, such as direct ELCs. Those same risks apply to the ITGCs upon which the automated information processing controls depend, along with the risk that the ITGCs will fail to support the ongoing effectiveness of the automated information processing controls.

• Use of a benchmarking strategy for automated information processing controls.

• Alternative sources of evidence that might be available to determine the continued operation of key automated information processing controls.

OAG Guidance

Audit evidence obtained about the implementation of an automated control may provide some level of evidence regarding the operating effectiveness of the automated control. When this evidence is considered in combination with evidence regarding the operating effectiveness of the Information Technology General Controls (from the date of implementation of the control to the current audit period), it may also provide substantial evidence about the automated control’s operating effectiveness during the relevant period.

Consequently for an automated control, the number of items required to be tested is generally minimal, assuming we have previously tested the control. This is because, where we are relying on any automated controls, or automated calculations, we will normally test Information Technology General Controls to be satisfied that the automated control / calculations continues to function properly. In the scenario where we do not have effective ITGCs, it may be more efficient to test automated controls using alternative techniques as described in the block below Automated controls approach with no ITGC evidence.

Regardless of whether we have effective ITGCs, we need to design the test prudently, checking that it addresses all critical iterations of the control. An iteration of a control is when an automated control is programmed to operate differently dependent upon the input. See the following examples of controls with multiple iterations:

Depending on the nature of the control and its risks, evidence of the operation of the automated control may be obtainable through sufficient inquiry, observation, examination, and/or reperformance procedures performed during a design evaluation of the related transaction process. It may be necessary to test more than one transaction to see how all important aspects of the control operate. Also, it may be more efficient or effective or both to test certain automated information processing controls (particularly more complex controls, calculations or reports) “through the application,” such as by directly examining a configuration setting in the application.

When deciding our testing approach for automated controls, we should consider whether

• we might obtain indirect assurance about the effectiveness of newly implemented controls through our tests of program change or program development controls (i.e., management’s process for testing of the operation of the control as part of their application implementation process);

• there is opportunity to use dual purpose testing (e.g., substantive tests in the audit of financial statements that might also provide point‑in‑time evidence about the controls over completeness and accuracy of a key report used in a manual control); and

• specialized skills are needed (e.g., to directly examine a configurable ERP setting or to evaluate the client’s approach to testing the control during application implementation).

The design of a test for an automated control considers the most relevant functionalities/attributes of the control, in order to assess the operating effectiveness of the automated control in all significant aspects, based on the information processing objectives that the control was designed to accomplish. The “test-of-one” concept is not limited to testing a single transaction handled by the automated controls. For example, consider the following:

• the data fields the application is programmed to process,
• the programmed calculation logic,
• whether the application has tolerance levels for automated instructions,
• the manner in which corrections are approved and processed,
• the manner in which exceptions are dealt with, and
• whether and how manual or systematic overrides are permitted.

Once we have gathered this information and performed enough work to evaluate the design of controls (including an understanding of the various control attributes and transaction paths), we then need to judge whether that work has also provided a sufficient level of assurance about whether the control is operating as designed. If not, additional procedures may need to be performed using one or more of the testing techniques described above. For example, it might be more efficient or effective to directly examine the configuration settings for the three‑way match in the application.

An example of where we might need to do more work to check that an automated control is operating as designed would be in testing the accuracy of an automated report used to generate a significant accounting estimate (e.g., an excess and obsolete inventory report supporting a reserve estimate). We could evaluate and possibly be satisfied with how the report fits into the design of controls by making inquiries of accounting and IT personnel regarding the nature of the data in the report, how it is accumulated, and how it is used in the accounting and estimation process. If the design of controls is such that the accuracy of the accounting estimate depends on the accuracy of the report and the accuracy of the report depends on proper programming in the application, then we would perform procedures designed to assess whether the report is complete and accurate according to its intended design.

Complex Automated Controls

For some complex automated controls, it might be impractical or even impossible to evaluate the design or the operating effectiveness of controls through typical “manual” procedures alone to understand a process by observing the flow of a transaction. An example might be an application that processes medical insurance claims. The automated process might use input data, edit checks, and attribute tests to direct transactions down alternative processing paths that are then subject to a variety of complex calculations to determine the actual claim value. The application might consider factors such as:

• Is the claim within the effective policy period?
• Is the medical service covered by the individual’s plan?
• Is the service provided in or out of network?
• Has the deductible been met?
• Is there a co‑payment required?
• At what location was the service performed?

Unlike a simple clerical process that can be easily re‑performed (such as the processing of sales invoice amounts as price times quantity), the variations and complexity of some processes (like this medical claims example) could require more sophisticated techniques for understanding and evaluating the transaction flow, the risks of misstatement, and the design and operation of controls.

OAG Guidance

When evaluating application-level security and access conditions (that is, determining “who has access to what”):

• Refrain from using sophisticated segregation of duties analysis tools prior to performing a risk-based evaluation of the specific security and access conditions that are important to the audit. Avoid generating and analyzing volumes of data on existing access rights and potential “conflicts.” Rather, first understand the risk of material misstatement of the financial statements associated with unauthorized or sensitive access rights (whether through fraud or inadvertent error), which is determined as part of our procedures to understand a process by observing the flow of a transaction or other procedures to understand what could go wrong and to identify related controls, and then focus our efforts specifically on those risks.

• Use the knowledge and information gained from past audits and from the work of the internal auditors to limit the amount of testing of security conditions in the current period, particularly when the ITGCs are effective and the environment is stable and not subject to significant change.

• Consider likelihood, not just magnitude, when evaluating the risks associated with privileged access (e.g., super-user / DBA access, or access to application utilities that could change financial data or records). Do not default to a fully substantive audit approach in the presence of IT deficiencies or exceptions without first considering the impact of these deficiencies on our audit approach (see OAG Audit 4028.4 for additional guidance on the impact of control deficiencies on our audit approach). The existence of privileged access is not a “de facto” deficiency in internal control. Reasonable professional judgment is needed to evaluate the risks associated with privileged access (both fraud risks and the risks of inadvertent error or data corruption), and the nature and extent of controls needed to mitigate those risks. See OAG Audit 5035.2.

OAG Guidance

In recurring audits we incorporate knowledge obtained during past audits into the decision-making process for determining the nature, timing, and extent of testing necessary. It is important to consider whether there have been changes in the controls or the process in which it operates. A benchmarking strategy is an efficient way to obtain evidence about the continued operating effectiveness of automated information processing controls. That is, if ITGCs are effective and continue to be tested and we can verify that an automated information processing control has not changed since it was last “baseline” tested, we can conclude the information processing control continues to be effective. After a period of time the baseline of the operation of an automated control would be reestablished. The length of time is a matter of professional judgment, considering factors such as

• the effectiveness of the controls in the IT control environment, including controls over application and system software acquisition and maintenance, access controls and computer operations;

• the auditor’s understanding of the nature of changes, if any, on the specific programs that contain the controls

• the nature and timing of other related tests;

• the consequences of errors associated with the information processing control that was benchmarked; and

• whether the control is sensitive to other business factors that may have changed. For example, an automated control may have been designed with the assumption that only positive amounts will exist in a file. Such a control would no longer be effective if negative amounts (credits) begin to be posted to the account.

OAG Guidance

There are a variety of techniques for testing whether an automated control operates effectively. They include:

1. Obtaining evidence that the automated control is operating effectively through sufficient inquiry, observation, examination, and/or reperformance procedures during our understanding of a process by observing the flow of the related transactions.

2. Running sample transactions (“test deck” or “integrated test facility”) through the application program or routine and comparing the output to expectations.

3. Replicating the output by running our own independent queries or programs on the actual source data.

4. Evaluating the logic of the application program / routine by

   • inspecting application system configurations,
   • inspecting vendor system documentation,
   • interviewing program developers (note that inquiry is not enough by itself),
   • inspecting the source code.

5. Testing the logic indirectly by performing substantive testing of the output to source documents or by reconciling it to independent, reliable sources (e.g., testing the accuracy of an aging report by comparing to circularization results or by tracing back to sales invoices).

When testing the operating effectiveness of an automated control consider the risks that: 1) the logic or rules programmed or configured into the application are not accurate to achieve the desired outcome, and/or 2) the application draws upon inaccurate or unreliable data, either through its sources or its timing of execution. For example, if we desire evidence that a systematic inventory costing calculation is properly valuing inventory in accordance with the entity’s stated costing convention (e.g., FIFO), our testing approach needs to provide evidence not only that the calculation is functioning as intended, but also that it is drawing from appropriately updated price and quantity data files at the right time in the processing cycle.

OAG Guidance

Examples of automated controls and suggested tests of control:

(*) Assumption: we have access to competent specialists who understand the database language used to write the query. In addition, proper ITGC audit procedures have been performed to obtain evidence over data integrity and reliability.