Greater transparency needed for NSW Government’s use of machine technology in decision-making

NSW Ombudsman Paul Miller has called for greater visibility of the use of artificial intelligence (AI) and other machine technology by NSW Government agencies.

In a report tabled in Parliament this morning, the Ombudsman has cautioned agencies that using machine technology in ways that do not accord with standards of lawfulness, transparency, fairness and accountability, could lead to findings of maladministration or potentially unlawful conduct.

The report, The new machinery of government: using machine technology in administrative decision-making, describes the increasing use of machine technologies in government decision-making processes. In NSW, agencies are known to be using machine technologies in the areas of fines enforcement, policing, child protection and driver licence suspensions.

Government agencies do not currently have an obligation to proactively report on their use of machine technology. They also do not routinely let people know when decisions affecting them are being made by or with the assistance of machines.

“The use of machine technology in the public sector is increasing, and there are many potential benefits, including in terms of efficiency, accuracy and consistency,” said Mr Miller.

“As an integrity agency, our concern is that agencies act in ways that are lawful, that decisions are made reasonably and transparently, and that individuals are treated fairly. Those requirements don’t go away when machines are being used.”

Complaints reveal extent of machine technology in NSW fine recovery

The NSW Ombudsman’s Office was prompted to write the report after becoming aware that the state’s debt-collection agency, Revenue NSW, was using an automated technology system to garnishee, or debit money from, the bank accounts of people who had failed to pay fines.

“My office began to receive a spate of complaints from people, many of them financially vulnerable individuals, who had discovered their bank accounts had been stripped of funds, and sometimes completely emptied,” Mr Miller said. “Those people were not complaining to us about the use of automation. They didn’t even know about it.”

”Following our intervention, and to its credit, Revenue NSW took a number of steps to address our concerns about unfairness.,” Mr Miller noted. “However, what it did not do – despite our suggestion that it should – is seek expert legal advice on whether the use of the automation process was lawful and in accordance with its powers under the Fines Act. Ultimately, we decided to seek that advice ourselves.”

The Ombudsman’s report annexes external legal advice, which concludes that the original automated process used by Revenue NSW to garnishee bank accounts was unlawful.

In response to the Ombudsman’s initial concerns, Revenue NSW modified its process in 2019 so that orders to take money from bank accounts ceased to be fully automated. However, the legal advice notes that these modifications do not appear to have fully addressed the legal concerns.

Going forward: practical steps, legislative authorisation and greater transparency

The Ombudsman’s report provides practical guidance to agencies on the proactive steps they should take to reduce the risk that machine technology could be unlawful or otherwise amount to maladministration.

These include:

• ensuring the design involves experts from fields other than IT, including legal advisors
• giving careful consideration to the relationship between the machine and the ultimate human decision-maker
• building in a rigorous pre-deployment testing and ongoing auditing regime, and
• taking action to ensure appropriate transparency.

“We are concerned that other agencies may also be designing and implementing machine technologies without appreciating all the risks, without transparency, and without getting appropriate legal advice’, Mr Miller said.

The report also suggests that Government agencies consider seeking Parliamentary approval through legislation before machine technology is adopted for important administrative functions.

‘Seeking express legislative authorisation not only reduces the risks for agencies,’ Mr Miller said, ‘it also gives Parliament and the public visibility of what is being proposed, and an opportunity to consider what other regulation of the technology may be required. That might include a statutory right to ask for a person to review any machine decision, or a requirement that the machine’s algorithms be externally validated, and then audited at regular intervals, with those reports to be made publicly available.’

Following tabling of the report this morning, the NSW Ombudsman’s Office is now going to engage with the NSW Government and local government sector to comprehensively map the use of machine technology in administrative decision-making processes across the state.

“Greater visibility is not a panacea to all of the potential issues that can arise when Government adopts machine technology,’ Mr Miller said. ‘But it is an essential starting point.’

• Download the full report