Guide for Performance Audits and Special Examinations on Learning, Training, and Development (November 2018)

Contents
Introduction
Purpose of this guide
Applicability and audience of this guide
Auditor’s responsibilities
Background and context
Key terms and concepts
Legislative and related requirements
Audit guidance
Why learning, training, and development is important
Identifying and assessing risks for audit selection and/or audit scoping
Recommended audit approach

Introduction

Purpose of this guide

The purpose of this guide is to assist audit teams who have identified a risk that employees in the entity under audit do not have the knowledge, skills, and/or competencies required to carry out their functions and meet organizational objectives. As a result, the audit team is considering and/or planning to carry out audit work in this area.

This guide provides audit teams with

  • an understanding of learning, training, and development, including legislative and related requirements;

  • guidance on determining whether learning, training, and development is an area of risk that should be examined; and

  • a recommended approach to auditing learning, training, and development. While this approach is comprehensive, audit teams may wish to tailor it to their audit context in consultation with the Human Resource Management Internal Specialist.

Applicability and audience of this guide

This guide applies to performance audits and special examinations. It is intended for engagement leaders and their respective audit teams.

Auditor’s responsibilities

Per the Direct Engagement Manual section 3081, audit teams are to consult with internal specialists when dealing with matters requiring specialized knowledge and experience.

Background and context

Key terms and concepts

Learning is the acquisition of new knowledge and ideas that change the way an individual perceives, understands, or acts.1

Training is an organized, disciplined way to transfer the knowledge and know-how that is required for successful performance in a job, occupation, or profession.2

Professional development is an activity that helps employees further their careers and is aligned with the organization’s business priorities.3

Learning, training, and development can be obtained formally through courses, workshops, and/or other events (e.g., conferences) developed in-house and/or by external service providers. They can also be obtained informally; for example, through on-the-job learning and coaching.

The extent of learning, training, and development offered by organizations may vary depending on their size and the experience of the employees that they recruit. For example, small organizations (those with fewer than 100 employees) may not have the resources to train and develop employees from within. They may rely more heavily on external service providers and/or they may recruit experienced employees who already have the knowledge and experience to carry out the work.

Legislative and related requirements

The legislative basis for learning, training, and development in the public service is the Financial Administration Act and the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development, which establish the responsibility of deputy heads to

  • determine the learning, training, and development requirements of employees;
  • fix the terms on which the learning, training, and development may be carried out; and
  • ensure that employees have the knowledge, skills, and competencies to do their work.4

To support the implementation of the Policy on Learning, Training, and Development, the Treasury Board of Canada Secretariat also developed a Directive on the Administration of Required Training.5

The people management component of the Treasury Board of Canada Secretariat’s Management Accountability Framework includes employee learning and development as a measure of organizational performance. The outcome of effective learning and development is “a skilled and agile workforce that has the competencies and flexibility to meet the needs of an evolving public service.” The Framework measures this through the percentage of employees in an organization who have documented learning objectives (a learning and development plan) as part of the performance management process. This information is collected and reported annually by the Treasury Board of Canada Secretariat.6

The importance of learning, training, and development was also referenced in the Clerk of the Privy Council’s Twenty-Fourth Annual Report to the Prime Minister on the Public Service of Canada. One of the Clerk’s three priorities is attracting, retaining, and developing top talent. The Clerk stated that the public service is experiencing a dramatic demographic shift as baby boomers retire in large numbers (9,000 retired in 2016) and that ensuring the public service recruits, develops, and supports the right people is now its “most pressing challenge.”7 Learning, training, and development will therefore be key to equipping the next generation of employees with the knowledge, skills, and competencies needed to carry out their work and achieve organizational objectives.

Audit guidance

Why learning, training, and development is important

As stated by the Treasury Board of Canada Secretariat, learning, training, and development “are key to ensuring that the public service is equipped to meet the challenges of the 21st century. The acquisition of skills and knowledge and the development of managerial leadership and know-how is critical for the effective management of the public service—it is the foundation of a responsive, accountable and innovative government.”8

If an organization’s workforce does not have the knowledge, skills, and competencies needed to carry out the work, this could

  • reduce employee productivity, which could lead organizations to place an over-reliance on other staff and/or more costly resources (such as overtime and contractors) to carry out the work;

  • hurt employee morale, which could lead to increased sick leave usage and employee turnover; and

  • in cases where physical risk is a key component of the job, threaten the well-being of employees and lead to increased accident rates and workers’ compensation claims.

Taken together, these consequences result in higher costs to complete the work and an unproductive, unstable, and unhealthy workforce—all of which may contribute to inefficient delivery of services to Canadians.

Identifying and assessing risks for audit selection and/or audit scoping

To help determine whether learning, training, and development is a risk that should be considered for audit, audit teams should ask the following questions:

  • Do employee survey results indicate problems with learning, training, and development?

  • Do exit interviews indicate problems with learning, training, and development?

  • Does the organization comply with the Treasury Board of Canada Secretariat’s Policy on Learning, Training, and Development9? Specifically:

    • Does the organization have a learning policy that supports its mission and mandate, and specifies requirements for job-related training and learning plans?

    • Does the organization identify and ensure employees receive mandatory training?

    • Do all employees have learning plans?

    • Does the organization meet the policy’s monitoring and reporting requirements?

  • Does the organization annually identify the learning, training, and development needs of its employees?

  • Does the organization commit the resources required to address its employees’ learning, training, and development needs?

  • Does the organization develop and implement learning, training, and development strategies and plans to address identified needs?

  • Does the organization monitor its learning, training, and development strategies and plans to ensure that employees who need the learning, training, and development receive it (in particular, mandatory training)?

  • Does the organization evaluate its learning, training, and development strategies and plans to determine whether they are achieving intended results, and make adjustments as required?

Audit teams should request and review relevant documents, such as

  • most recent employee survey results;

  • summary of exit interviews;

  • learning, training, and development policies, directives, and guidelines;

  • learning, training, and development strategies and plans;

  • list of learning, training, and development needs;

  • reports and analyses on learning, training, and development provided (e.g., learning, training, and development needed versus taken, particularly for mandatory training);

  • evaluations and analyses that assess the effectiveness of learning, training, and development provided;

  • corporate risk profile and risk register;

  • reports to senior management on the organization’s learning, training, and development.

Recommended audit approach

The tables below outline the audit criterion and sources, as well as the audit questions, key interview questions and documentation, and audit steps and techniques recommended to carry out the examination work on learning, training, and development.

Audit Criterion and Sources

Audit criterion: The organization ensures that its workforce has the knowledge, skills, and competencies needed to carry out required work and meet objectives.

Sources

NOTE: For audit entities that are not subject to Treasury Board of Canada Secretariat policies, refer to the organization’s own learning, training, and development policy and directives for criterion sources.

Financial Administration Act, last amended 2016 (http://laws-lois.justice.gc.ca/eng/acts/f-11/)

“Powers of deputy heads in core public administration”

(…)

“12 (1) Subject to paragraphs 11.1(1)(f) and (g), every deputy head in the core public administration may, with respect to the portion for which he or she is deputy head,

(a) determine the learning, training and development requirements of persons employed in the public service and fix the terms on which the learning, training and development may be carried out;”

“Powers of other deputy heads”

(2) Subject to any terms and conditions that the Governor in Council may direct, every deputy head of a separate agency, and every deputy head designated under paragraph 11(2)(b), may, with respect to the portion of the federal public administration for which he or she is deputy head,

(a) determine the learning, training and development requirements of persons employed in the public service and fixing the terms on which the learning, training and development may be carried out;

Treasury Board of Canada Secretariat Policy on Learning, Training, and Development, last amended 2008 (https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405)

5. Policy Statement
(…)
“5.2 The expected results of this policy are that
(…)

  • employees at all levels will acquire and maintain the knowledge, skills and competencies related to their level and functions;”

6. Policy Requirements

“6.1 Deputy heads are responsible for ensuring that:

  • their employees are informed of this policy and have the knowledge, skills and competencies to do their work;”
Audit Question Key Interview Questions and Documentation Audit Steps and Techniques

Q1—Does the organization have a learning, training, and development policy?

Key interview questions

  • Does the policy meet the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development? Specifically, does it specify

    • requirements for job-related training?

    • requirements for learning plans?

    • how it will meet monitoring and reporting requirements?

  • Does the policy clearly define the roles and responsibilities for learning, training, and development?

  • Are roles and responsibilities for learning, training, and development communicated to those responsible, and if so, how?

Key documentation

  • Learning, training, and development policy; directives; guidelines; procedures

  • Communications to staff regarding roles and responsibilities for learning, training, and development

Request and review documentation.

Interview

  • Officials responsible for learning, training, and development; e.g., VP HR; Learning, Training, and Development Director / Manager; Training Coordinator

  • Senior / line managers (if applicable)

Assess whether the organization has a learning, training, and development policy that meets the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development, and whether roles and responsibilities are communicated to those responsible for learning, training, and development.

Document observations and respond to audit question.

Q2—Does the organization annually identify the learning, training, and development needs of its employees?

Key interview questions

  • How are learning, training, and development needs identified (e.g., performance management process, business planning, risk assessments)? Is this an annual exercise?

  • Are learning plans developed for each employee? Is this monitored? (Learning plans are required by the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development.)

  • Are needs documented and rolled up to the organizational level for planning purposes? If so, how?

  • How does the organization prioritize and determine what needs will be met?

  • Does the organization identify mandatory training, and if so, does it identify employees who need it? (Organizations must identify mandatory training per the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development.)

Key documentation

  • Learning, training, and development policy; directives; guidelines; procedures

  • Documentation that explains the performance management process, and specifically, how learning, training, and development needs are identified

  • Completion rate of employee learning plans for the last two fiscal years. (Organizations are required to submit this information to the Treasury Board of Canada Secretariat as part of the Management Accountability Framework.)

  • Documentation and analyses that summarize employee learning, training, and development needs for the last two fiscal years

  • Documentation that explains what needs will be met from those identified, and why

  • List of mandatory learning, training, and development by employee level

  • Other relevant documentation, e.g.

    • senior management committee documentation (agendas, minutes, presentations and reports discussed, records of decision);

    • HR committee meeting documentation;

    • relevant internal audits or evaluations of the organization’s learning, training, and development program and systems and practices.

Request and review documentation.

Interview

  • Officials responsible for training and development; e.g., VP HR; Learning, Training, and Development Director / Manager; Training Coordinator

  • Senior / line managers (if applicable)

Assess whether the organization identifies the learning, training, and development needs of its employees, in particular, mandatory training, and rolls them up and prioritizes needs to be addressed.

Document observations and respond to audit question.

Q3—Does the organization develop and implement strategies and plans to address the learning, training, and development needs identified?

Key interview questions

  • Are learning, training, and development strategies and plans aligned with strategic and operational priorities?

  • Does the organization have a training coordinator or similar position that is responsible for supporting the implementation of mandatory training? (This is required by the Treasury Board of Canada Secretariat Directive on the Administration of Required Training.)

  • Does the organization develop and deliver its own learning, training, and development or does it rely on external service providers for this (make or buy)?

  • How does the organization make this decision (e.g., resource availability, cost/benefit analysis, proprietorship, sensitivity of the training)?

  • Does the organization’s plan for learning, training, and development

    • take into account the resources (human, financial) needed to implement it;

    • identify targets, accountabilities, and timelines; and

    • consider official language requirements?

  • Does the organization deliver planned learning, training, and development in a timely manner?

Key documentation

  • Learning, training, and development policy; directives; guidelines; procedures

  • Learning, training, and development strategies and plans for the last two fiscal years

  • Learning, training, and development delivered during the last two fiscal years

  • Documentation that explains how learning, training, and development are developed and delivered (e.g., in-house versus external service provider)

  • Documentation that explains the organization’s budget for learning, training, and development activities and the amount spent on learning, training, and development for the last two fiscal years

  • Other relevant documentation, e.g.

    • senior management committee documentation (agendas, minutes, presentations and reports discussed, records of decision)

    • HR committee meeting documentation

    • relevant internal audits or evaluations of the organization’s learning, training, and development program and systems and practices

Request and review documentation.

Interview

  • Officials responsible for training and development; e.g., VP HR; Learning, Training, and Development Director / Manager; Training Coordinator

  • Senior/line managers (if applicable)

Assess whether the organization

  • has a required training coordinator;

  • develops strategies and plans that are aligned with strategic and operational priorities and address its employees’ learning, training, and development needs;

  • commits the needed resources (human, financial) to implement them;

  • considers official languages in the delivery of learning, training, and development; and

  • delivers the learning, training, and development planned in a timely manner.

Document observations and respond to audit question.

Q4—Does the organization monitor its learning, training, and development plans to ensure that employees who need planned learning, training, and development receive it?

Key interview questions

  • Does the organization track attendance at courses, workshops, and/or other learning, training, and development delivered, particularly for mandatory training?

  • Does the organization assess whether employees who required the learning, training, and development received it? (Organizations must track the successful completion of mandatory training per the Treasury Board of Canada Secretariat Directive on the Administration of Required Training.)

  • For employees who did not receive the learning, training, and development planned, does the organization examine the reason(s) why (e.g., course not offered in employee’s official language) and the impact?

  • Is information on employee learning, training, and development needs and the learning, training, and development taken by employees collected and maintained, particularly for mandatory training? If so, how (e.g., spreadsheet, HR information system)? (Organizations must track the successful completion of mandatory training per the Treasury Board of Canada Secretariat Directive on the Administration of Required Training.)

  • Does the organization have data quality systems and practices in place to ensure the integrity of HR data, including learning, training, and development data?

  • Are results reported to senior management?

Key documentation

  • Documentation and analyses that shows employees who have been identified for learning, training, and development, particularly for mandatory training

  • Documentation and analyses that show employees who have successfully completed the learning, training, and development, particularly for mandatory training

  • HR information system documentation (e.g., user manuals, guides, data quality reports) where learning, training, and development data is collected and maintained

  • Reports to senior management on learning, training, and development planned and taken, particularly for mandatory training.

  • Other relevant documentation, e.g.

    • senior management committee documentation (agendas, minutes, presentations and reports discussed, records of decision)

    • HR committee meeting documentation

    • relevant internal audits or evaluations of the organization’s learning, training, and development program and systems and practices

Request and review documentation.

Interview

  • Officials responsible for training and development; e.g., VP HR; Learning, Training, and Development Director / Manager; Training Coordinator

  • Senior / line managers (if applicable)

Perform a walkthrough of key HR information systems (electronic or paper-based) that collect and maintain learning, training, and development data to determine what information exists, its quality, and how it is used and reported on, if this was not already done in the planning phase.

Assess whether employees who were identified as needing planned learning, training, and development received it (particularly for mandatory training). If they did not, assess the impact on the employee and on the organization’s ability to deliver the required programs and services to Canadians.

Document observations and respond to audit question.

Q5—Does the organization evaluate its learning, training, and development strategies and plans to determine whether they are achieving intended results, and make adjustments as required?

Key interview questions

  • Does the organization have a learning, training, and development strategy, and if so, what does it entail?

  • How does the organization assess the effectiveness of its learning, training, and development activities? Does it use:

    • course evaluations?

    • tests and exams on training material?

    • employee performance on the job (e.g., performance appraisals)?

    • other?

  • How does the organization evaluate its learning, training, and development strategies and plans to determine whether they are meeting the needs of employees? How often is this done?

  • Is information from this evaluation used to adjust learning, training, and development strategies and plans as required?

  • Are the results reported to senior management?

Key documentation

  • Learning, training, and development evaluation strategy.

  • Documentation that explains how learning, training, and development activities are assessed (e.g., course evaluation sheet, tests and exams administered, on-the-job assessments).

  • Reviews of learning, training, and development strategies and plans for the last two fiscal years.

  • Reports to senior management on the review of learning, training, and development strategies and plans, including any improvements and adjustments proposed.

  • Other relevant documentation, e.g.

    • senior management committee documentation (agendas, minutes, presentations and reports discussed, records of decision);

    • HR committee meeting documentation;

    • relevant internal audits or evaluations of the organization’s learning, training, and development program and systems and practices.

Request and review documentation.

Interview

  • Officials responsible for training and development; e.g., VP HR; Learning, Training, and Development Director / Manager; Training Coordinator

  • Senior / line managers to obtain their views on whether learning, training, and development needs are being met (e.g., do they feel that their employees have the knowledge, skills, and competencies needed to do their work?)

Assess whether the organization

  • has a learning, training, and development evaluation strategy and

  • evaluates its learning, training, and development strategies and plans to determine whether they are achieving intended results, and makes course corrections as required.

Document observations and respond to audit question.

1 Treasury Board of Canada Secretariat Policy on Learning, Training, and Development: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405

2 Treasury Board of Canada Secretariat Policy on Learning, Training, and Development: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405

3 Treasury Board of Canada Secretariat Policy on Learning, Training, and Development: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405

4 Financial Administration Act: http://laws-lois.justice.gc.ca/eng/acts/f-11/; Treasury Board of Canada Secretariat Policy on Learning, Training, and Development: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405

5 Directive on the Administration of Required Training: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12407

6 Management Accountability Framework: https://www.canada.ca/en/treasury-board-secretariat/services/management-accountability-framework.html

7 Clerk of the Privy Council’s Twenty-Fourth Annual Report to the Prime Minister on the Public Service of Canada: https://www.canada.ca/en/privy-council/corporate/clerk/publications/2017-annual-report-landing-page.html

8 Treasury Board of Canada Secretariat Policy on Learning, Training, and Development: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12405

9 Note that the Treasury Board of Canada Secretariat Policy on Learning, Training, and Development applies to the core public administration as defined in the Financial Administration Act; i.e. the organizations named in Schedule I and the other portions of the federal public administration named in Schedule IV of the Act. The Policy does not apply to separate agencies or Crown corporations. However, it indicates that these organizations may use the Policy to develop their own policies.