Implementing automated decision-making systems

What to consider when introducing or reviewing automated decision-making (ADM) tools in your agency.

Whenever an agency is considering changing its decision-making processes or technology, it is important to recognise that its situation will be legally unique. Agencies will need to pay close attention to the particular statutory function and the legislation that governs it.

Oversimplifying the implementation of ADM systems, or treating it as merely a ‘technology project’ is a major risk. If your agency is considering adopting new ADM systems or is reviewing how it uses those technologies, follow these steps.

Step 1: Assemble the right team

Implementing ADM systems is not only an IT project. It should be an agency-wide initiative that involves a team of lawyers, policymakers, and operational and technical experts.

Each expert should:

  • have clearly defined roles and responsibilities in the project
  • be involved in the project from the start
  • have an appropriate level of authority.

An obvious risk is that legislation and associated policy and guidance may be misinterpreted when it is translated into code. Establishing the right team from the start will help ensure the technology is designed by people who are best placed to assess whether it will be delivering lawful and fair decisions. This is more efficient and safer than trying to incorporate feedback from experts on a project that’s almost complete.

Step 2: Determine the role of humans

Deciding how far a process can be automated is not an easy question. It needs to be assessed in the context of the agency’s function and legislation. For example, discretionary decision-making requires human involvement and it should be assumed that it can never be fully automated – see Who can make decisions.

Assessing whether there is enough genuine human involvement in an automated process involves a range of factors. For example, the person actually making a decision may need:

  • enough time to consider the outputs of the ADM system and any individual circumstances or other sources of information
  • access to the source information the ADM system used and any other relevant information
  • an understanding of how the ADM system works
  • the ability to provide comprehensive reasons about the role of the ADM system
  • the ability to make or request changes to the ADM system
  • the authority and practical ability to reject the outputs or override a decision of the ADM system.

There must be the required amount of human involvement in practice, not just on paper.

Agencies need to be wary of ‘technology complacency’ – the tendency for decisions to become increasingly (and potentially unlawfully) controlled by the ADM system’s outputs over time.

Step 3: Ensure transparency

Identify how your agency will provide reasons for decisions made using ADM systems early in the project.

When an agency gives reasons to an individual affected by a decision, those reasons must be meaningful. Among other things, a meaningful explanation should include:

  • that automation was involved
  • the extent to which automation was used
  • what information is processed by the ADM system
  • the date and version of any technology
  • how (in lay terms) the technology works.

The above is in addition to the usual requirements for giving reasons. For more information, see Administrative Law and automated decision-making.

Step 4: Test early and often

Test technology before going live. Test and review at regular stages once it’s in operation to ensure decisions are accurate and legal. Regular testing and monitoring reduces technological, legal and performance risks.

The type of testing required will depend on why and how you’re using ADM systems. Agencies need to dedicate time and resources to conduct independent audits and testing. This includes:

1. Legal testing

A thorough legal audit that looks at how the law has been interpreted into code. Lawyers conducting the legal audit should not be the same people who helped design the technology.

2. Validation and accuracy testing

Verify and validate machine outputs against human decisions. This should be conducted both at the start and regularly while ADM systems are in use.

3. Testing for algorithmic bias

Data sets may contain information with historical biases or inequalities, which can be amplified by ADM systems.

Agencies need to embed testing procedures that address bias in their risk management strategy for ADM systems.

Monitor and review

A monitoring and review process for another agency’s technology won’t necessarily work for yours. The process needed will depend on your technology, the risks that your agency may face, and your agency’s needs.

A robust auditing process should be established before implementing any technology. It should define:

  1. who is responsible for audits or reviews
  2. the scope of information and data to be reviewed
  3. a mechanism for monitoring the progress of any recommended changes.

To help you establish business-as-usual monitoring, consider the following:

  • Regular reviews – the schedule should be sustainable and report on risk analysis and mitigation.
  • Routine certification, testing and auditing – this should be undertaken by an independent expert.
  • Systematic review of identified errors – using a group of errors collected by users, both false positives and false negatives should be reviewed.
  • Audits of ADM system outputs – this may be embedded into the agency’s existing quality assurance processes. Consider this as part of other information you receive such as complaints and feedback from staff.
  • Random auditing of cases – regularly hold a sample of cases for human decision-making, independent of the ADM system.

It is important to transparently identify how your ADM systems will be used and monitored. Embedding these processes early will help to ensure accountability. It may also be required if the technology is being subject to an external review by the NSW Ombudsman or other oversight body.

Step 5: Consider legislative amendment

It’s unlikely that the legislation related to your agency’s functions explicitly deals with the use of ADM technology.

Agencies working with lawyers and policymakers can take steps to determine whether ADM systems can be used appropriately within the existing legislation, or whether the legislation should be amended. If the legislation is to be amended to explicitly authorise the use of the ADM, consideration should also be given to making amendments relating to issues such as testing, auditing, transparency and accountability.

More information

To learn more about the design, implementation and review of ADM systems: