Unbiased measurement & assessment

PGPA Rule section 16EA(c)

The performance measures meet this requirement when they provide an unbiased basis for the measurement and assessment of the entity's performance.

Performance measures must provide an unbiased basis for assessment of the entity’s performance.

When designing an approach to collecting and/or analysing information, entities need to consider how bias might be unintentionally introduced.

Common sources of bias

Exclusion bias

Occurs when relevant information is not collected (for example, where information is not collected on activities that make key contributions to fulfilling a purpose).

Sampling bias

Occurs when individuals or groups are disproportionately represented in the sample from which information is collected (for example, when key stakeholders are omitted from the sample group).

Interaction bias

Occurs when the sample group is aware that it is being observed/tested and changes its behaviour either consciously or subconsciously (for example, when people perform differently because they know they are being observed/tested).

Perception bias

Occurs when the people collecting or analysing information have preconceived ideas about how a system should behave or about what the results will be.

Operational bias

Occurs when the process for collecting information is not followed or when errors are made in the recording and analysis of data.

Case studies

Where performance measures are based on case studies or survey results, entities should ensure that measures are supported by clear methodology that explains the basis for selecting case studies and identifies how surveys will be conducted.

Case studies selected ‘after the fact’ (that is, after a particular activity has begun or has been completed, or after the entity’s period for performance has ended) introduce the potential for bias.

Case studies should not be relied upon as a stand-alone measurement unless the scope of the case study is predetermined, the activities clearly stated, and the measurement methods determined in advance. This avoids the risk of introducing bias where only favourable case studies that tell ‘success stories’ are selected.

Case studies and details of information to be collected should therefore be decided at the time the corporate plan is developed and before information collection occurs. Sufficient information should be included in the corporate plan to provide confidence to the reader that the selection of case studies and reviews are unbiased.

Surveys and contracted researchers

Where performance measures are based on survey results, documentation should be created which identifies factors including the survey sample population and size, how the survey will be conducted (including controls over the survey process and data) and an acceptable survey response rate.

Where performance measures are based on self-reporting by a third party provider of services, an assessment of the risk of the provider skewing results to be positive in nature should be undertaken. This would include assessing how this risk will be addressed and disclosed.

Where performance measures are based on assessment and measurement undertaken by those responsible for the management of the program or activity, there should be adequate controls in place to demonstrate that the result was objectively assessed. These might include ensuring the process of selecting which components or client groups the program or activity is targeting is independent or involves a random selection process. It might also include ensuring independence of assessment through including individuals not involved in the management of the program or activity in the assessment process or using a totally independent assessment process.

When considering the use of a contracted research and survey provider the selection process should take account of the qualifications and experience of the provider and the standards under which the provider will conduct work.

This includes ensuring the work is conducted in accordance with:

international quality standards (for example ISO 20252 Market, opinion and social research, including insights and data analytics — Vocabulary and service requirements)
information security standards (for example ISO 27001 Information security management)
Australian Privacy Principles (Commonwealth Privacy Act 1988).

A statement of quality and compliance with standards should be provided in the research report.