Element 5: Control Effectiveness

This information sheet is intended to assist Commonwealth officials at the Specialist and Executive levels understand:

• the different responsibilities for control owners, risk owners and treatment owners
• different types of controls and critical controls
• how to design effective controls
• how to establish a strong overall control environment.

A control is any process, policy, device, system, practice or other action that is put in place to modify the likelihood or consequence of a risk or to detect if a risk is happening. To assess whether your control is effective, you need to consider both the design and implementation of the control. It is important that controls are tested regularly to determine whether the control is effective in mitigating and managing the risks as expected. Controls do not operate in isolation. A strong overall control environment provides assurance to an entity that its objectives will be achieved and that the risk management system is performing.

• Risk Owners: Are accountable for managing, monitoring, reporting and escalating risks
• Control Owners: Are responsible for implementing and maintaining effective controls, including assessing their effectiveness and monitoring and reporting on performance. Control Owners are often responsible for reporting to Risk Owners on their control. Depending on the criticality of a control and the maturity of an entity’s risk management function, an internal or external stakeholder can undertake a control effectiveness review.
• Treatment Owners: Are responsible for implementing and monitoring treatments where the controls in place are ineffective and further mitigation activities are required.

Controls will have an impact on a risk. They usually fall into one (or more) categories:

• Preventative: These controls reduce the likelihood of a causes of the risk occurring. Examples include procedures, delegations, policies, system controls, and training.
• Detective: These controls identify failures in the risk management environment and help you identify if the risk has occurred. Examples include reconciliations, exception reporting, investigations, performance reviews and staff surveys.
• Corrective: These controls mitigate the consequence and/or rectify a failure after it has been discovered. These include business continuity plans, continuous improvement actions, crisis management and disaster recovery plans.

In most cases, controls will not be new items to be put in place just to manage the risk; controls will usually exist for other reasons. For example, many entities have multi-factor authentication in place to address one of the criteria within the Australian Signals Directorate’s Essential Eight, which is a compliance requirement, but it also addresses cyber security risk.

For most risks, it is expected that most controls will be preventative, with some detective and some corrective controls. Some controls may fall into multiple categories. For example, security training may be one of your key preventative controls for a personnel security risk, but it could also be a corrective control if you implement more training after a breach.

One of the common mistakes is to assume everything that your team does is a control. It is important to ask:

• Does it prevent or minimise a cause of the risk?
• Does it affect a consequence of the risk?
• Does it help you identify if a risk is happening?

If the answer is ‘no’ to all 3, then it is likely it is not a control to your risk.

Critical controls are those controls that are so crucial to preventing the risk from happening or mitigating the consequences of the risk event. Even though other controls may exist, if a critical control were to fail, it would significantly increase either the likelihood or the consequence of the risk.

Questions to ask to determine if a control is critical include:

• Does the control represent a significant barrier to the risk event or prevent consequence severity?
• Is the control the only barrier or layer of protection preventing the risk event?
• Does the control prevent a number of threats or mitigate multiple consequences?
• Does the control operate independently to other controls?

Controls should be proportionate and commensurate with the nature of the risk being managed and the subsequent consequence. They should also be reflective of the size, nature and risk profile of the entity. Larger entities with more resources at their disposal can invest in more extensive and complex controls to moderate particular risks.

When designing a control, consideration of the entity’s risk management maturity and risk environment should be factored into how the control is constructed.

Sometimes a new control will need to be designed if the existing controls for the risk are inadequate. Considerations when designing a control include:

Knowing if controls are effective is crucial in determining if the risk is being managed well. When assessing the current level of the risk, you need to take into consideration the controls that currently exist and how effective the controls are.

Some people use their ‘best guess’ to pick a control effectiveness rating. This is usually not accurate and can be affected by a number of biases. The most effective way to determine control effectiveness is to develop a regular testing program for the control based on documented evidence.

The approach to control testing should look at both the design and implementation of a control. Controls could be well-designed but could be implemented badly. For example, on the surface, a policy may be a great preventative control, but if business areas do not adhere to the policy, then it is not an effective control. Further information on who should undertake control effectiveness testing and approaches to testing is provided in the below sections.

This regular testing program can be recorded in something called a Control Profile. These should be tailored for each agency, but usually include information such as:

• Name of control
• Purpose / objective of the control
• Relevant risk
• Control owner
• Activities to achieve control objectives
• For each activity:
  • Method of testing
  • How often
  • Responsible person (for testing)

Controls can be tested in a variety of different ways, which are not mutually exclusive. The best way is to be able to ‘sight’ the work being done or the process or system being in place. This becomes more important where the control is critical and the severity of the risk is higher. By itself, reviewing policy or process documents or interviewing staff has limited value in providing assurance. Formal approaches are preferred where the nature of the risk is more severe; however, usually it is more cost effective to implement the informal checks.

The following are some common methods of control effectiveness testing:

Who should undertake control effectiveness testing?

There are a variety of different stakeholders across the 3 Lines Model who are able to undertake a control effectiveness review, depending on the criticality of the control and the complexity of the testing. For each control, there should be an assessment of who is best placed to test the control and how often it should be tested.

For example:

• The control owner: Staff at the operational level (‘first line’) who are responsible for the control are able to undertake an assessment of the effectiveness of the control.
• Framework owners: Officials at the second line are also able to undertake control effectiveness review.
• Control advisory team: Depending on the maturity of the entity, a dedicated control advisory team could be engaged to undertake a review of a control/system of controls. This could include a mixture of senior leadership and control owners who have a substantive knowledge and understanding of the processes in place across the entity.
• Peer reviews: External parties within the industry can be used to conduct a review of an entity’s system of controls. They can bring an independent set of eyes that objectively assesses the operating ability of the control.
• The internal audit function: An entity can engage an internal audit to validate and test the effectiveness their controls. This is part of the ‘third line’ of assurance and can be useful in order to provide an additional line of sight and review over the functionality of an entity’s system of controls.
• Regulators: In the case where there may be a cause for concern over the risk management approach of an entity, regulating bodies may be engaged to review the effectiveness of an entity’s system of controls. This is not a preferred approach and could be adopted in order to provide a level of assurance that the entity is meeting compliance standards.

A system of control refers to a group of controls, as opposed to individual controls, within an entity. Controls often do not work in isolation but instead operate within a system of controls that can more effectively mitigate a particular risk. For example, to test the effectiveness of your car’s safety corrective controls it would be necessary to review the quality of airbags, Electronic Stability Control (ESC), brakes and other safety mechanisms as a system of controls. This is as opposed to evaluating the safety of the car by only assessing one of these controls individually.

It is important that entities establish and build their system of controls around an assurance strategy that considers and coordinates review by means of supervision, management review and internal audit. Whilst it is the responsibility of control owners and risk owners to monitor the effectiveness of controls, Accountable Authorities are able seek advice from their Audit Committee over the adequacy of this broader assurance strategy.

Some entities use an annual declaration and assurance to the Accountable Authority that the following is occurring in relation to an entity’s system of control:

• The systems and resources that are in place for identifying, measuring, evaluating, monitoring, reporting and controlling or mitigating material risks, and the risk management framework, are appropriate to the institution, having regard to the size, business mix and complexity of the institution
• The risk management and internal control systems in place are operating effectively and are adequate having regard to the risks of the entity they are designed to control and
• The Risk Management Strategy has complied with each measure and control

Characteristics of a good system of controls include the following:

• The controls in place can be relied upon to prevent a risk materialising
• The controls in place are formally documented, current and understood by staff
• Ownership for the controls is clearly defined and understood
• Effectiveness of controls are formally reviewed and monitored by responsible management on a regular basis

Management ensures that the entity’s processes are operating in accordance with controls in place.

STEP 1: First of all, it is important to understand the control’s purpose. This can involve understanding what risk the control is intended to mitigate or manage, as well as whether the control is preventative, detective or corrective. This will enable you to determine the intended effect of the control and the causes or consequences it seeks to manage. You should also identify what inputs will be involved in testing the control in order to develop an understanding about it. For example, for an e-learning training module, the input for testing the effectiveness might be the percentage completion rate if the module by employees.

STEP 2: It could be necessary, depending on the nature of the control, to gather evidence to test the control. This could involve obtaining data and documentation that outlines whether the control is having its intended effect. This evidence could be gathered through a myriad of different ways, including conducting a survey, undertaking modelling, reviewing errors and incidents or looking at quality control.

STEP 3: Once the evidence is gathered, the next step is to conduct an evaluation of the effectiveness of the control. A ratings system allows for the design and the operating effectiveness of the control to be assessed. A three-tier scale of control effectiveness involving the control being deemed effective, partially effective or ineffective offers the most simple way in which to create a common understanding around the effectiveness of the control.

STEP 4: After evaluation, it can be determined whether a control needs to be updated if it is operating ineffectively. Alternatively, a risk treatment can be undertaken in order to mitigate any residual risk that is outside of the entity’s tolerance level. This process allows for entities to undertake any new actions whilst also providing a level of assurance if the control is operating effectively.

STEP 5: After this step, a risk register can be updated or populated to reflect the control effectiveness review.

This case study is intended to assist Commonwealth officials at Specialist and Executive levels understand:

• what is a control and how to measure control effectiveness
• practical tips to review and strengthen controls

This case study can be useful to entities wanting to take a structured approach to assessing control effectiveness.

This case study provides guidance for entities seeking to develop or formalise their approach to reviewing the effectiveness of controls, as modelled off the methods used by the Australian Financial Security Authority (AFSA). AFSA is an executive agency in the Attorney-General’s Portfolio responsible for Australia’s personal insolvency and personal property securities systems. AFSA’s management board recognised an opportunity to improve the agency’s risk management approach in order to embed more effective mitigation of risk at all levels of the agency. AFSA invested in a contemporary approach to risk management that focuses on utilising objective sources of evidence to verify the actual effectiveness of critical actions, activities and measures that help controlled key agency risks. This approach has been beneficial to AFSA as it has streamlined their enterprise risk reporting, and has created a proactive agency risk culture.

A control is something that regulates or modifies the realisation of, or response to, a risk. Controls take many forms, including any process, policy, device, practice or some other action which modifies risk. They can be preventative, detective or corrective in nature.

The realisation of a risk event is often a result of ineffective, poorly implemented or untested controls rather than unknown risks. The key to successfully managing risk is therefore to ensure controls are effective. Control effectiveness describes how well a control manages the risk it is meant to modify. Entities should periodically review control performance against its purpose and anticipated outcomes and determine whether the control remains suitable to support achieving the objectives of the entity.

Review and stocktake existing risk controls

AFSA reviewed its existing risk management framework and identified that it was largely compliance focused, and burdened staff through the completion of overly detailed and complicated risk registers. This created a culture where staff viewed risk management as a technical, paperwork driven activity that had little connection to their day-to-day work. Staff feedback identified that the risk management system was unnecessary complex, risk registers had a high potential for user error, and there was significant duplication of risks and controls across multiple registers. This meant that agency governance bodies could not have meaningful discussions regarding risk, and risk was not clearly linked with the entity’s Corporate Plan.

Evidence-based risk management

The agency decided to invest in a contemporary approach to risk management that instead focused on utilising objective sources of evidence to verify the actual effectiveness of critical actions, activities and measures that help control key agency risks. Whilst there may be numerous controls that can be applied against a risk, critical controls are the most important prevention, detection and mitigation controls. By focusing only on critical controls and ensuring that objective sources of evidence can be used to regularly verify their effectiveness, there is greater confidence that risks are actually effectively managed.

As a result, AFSA decommissioned its risk registers and replaced them with a series of critical control profiles and risk bowties. This approach uses high-level diagrams to summarise key risk information in an accessible form, and enables a quick visual analysis of control effectiveness and therefore risk exposure. The use of streamlined visual templates increases the focus on control effectiveness and identifies where further effort or investment can be targeted to address control gaps.

Ongoing review of control effectiveness

Each critical control is assigned to a control owner, who has the role of actively working across the agency to utilise objective sources of evidence to assess the effectiveness of their control. This enables AFSA to maintain ongoing board engagement with risk by utilising near misses, realised risks event or fictional yet plausible scenarios to drive a rolling program of risk walkthroughs, informed by evidence-based control effectiveness assessments. This process brings together the risk owner, control owners and board members to actively explore control gaps and current-state preparedness, and discuss areas for enhancement. AFSA has extended this approach across the agency and encourages staff to participate in similar conversations through risk forums. These forums have become valuable opportunities to provide assurance to the board that strong controls are in place while also creating opportunities for staff to identify areas of improvement.

This engagement has been critical to the success of AFSA’s approach, and it provides a clear line of sight between senior executives and the actual staff members undertaking the critical actions and activities that manage the risk. This is a level of transparency and confidence that can be difficult to achieve using other risk management approaches.