Businesses warned to understand their AI systems as countries start to regulate

Governments around the world will be taking a closer look at tackling evolving artificial intelligence regulation in 2022 as the pandemic’s labour shortage pushes for automation – which could have implications for Kiwi businesses 

As new technology allows for advances in artificial intelligence, regulators and businesses continue their struggle to keep up interventions to ensure people’s interests are protected.

Bell Gully senior associate Kristin Wilson says artificial intelligence (AI) is an area that is developing rapidly and many businesses don’t fully understand the AI software they’re using. 

She says Covid fast-forwarded uptake of AI technology, raising concerns about bias and privacy.

“In light of Covid, AI has done all sorts of interesting things, from allowing more effective diagnosis and treatment, to contact tracing. But on the other side of things there have been potentially negative developments. There’s been questions raised around AI used by social media platforms spreading misinformation and disinformation.”

In 2018, Immigration NZ was criticised for using AI to prioritise the deportation of overstayers based on factors such as race. Immigration minister at the time Iain Lees-Galloway said he wasn’t aware of this and the programme was shut.

Then last year the New Zealand Police adopted US technology company Clearview’s surveillance software, which was found to be bad at recognising certain ethnicities.

In 2020 the Government introduced the Algorithm Charter, which is currently being reviewed. 

The Algorithm Charter is a commitment by government agencies to manage their use of algorithms in a fair, ethical, and transparent way. 

“It’s all very principle-based. But there’s no agency that feeds these apps and monitors any of this. It’s all fairly voluntary, there’s no enforcement mechanism. So there isn’t a real stick to it. So there’s no sort of strict regulation,” Wilson says.

The European Union’s proposed AI regulation is intended to be extraterritorial like its General Data Protection Regulation privacy legislation which was introduced in 2018.

The regulations would apply to any AI system used or providing outputs within the European Union, signalling implications for New Zealand businesses too.

The EU’s AI draft regulations divide systems into three categories: unacceptable-risk AI systems, high-risk AI systems, and limited- and minimal-risk AI systems. Businesses will be able to use this framework as a starting point in developing their own internal risk-based taxonomies.

Penalties for breaching AI regulation could amount to a fine of up to 4 percent of the company’s worldwide turnover, or 20 million euro.

And if a business uses AI that has an unacceptable risk, that’s 6 percent of worldwide turnover at 30 million euro. 

Wilson says it’s important for businesses to understand the AI software they’re implementing. 

“Make sure that your AI use is recorded properly, so that it can be comprehensively described and easy for consumers to understand.”

NZ Law Foundation law and emerging technologies chairman Colin Gavaghan has been working on a three-year project exploring the implications of artificial intelligence for New Zealand. 

His research along with an AI expert and a philosopher looked at the government’s use of AI.

Gavaghan says general purpose laws that apply to algorithms apply to algorithmic decisions, just the same as they apply to human ones.

But the problem is that if you can’t understand how the algorithm has reached this decision, it’s difficult to use the laws. 

The nature of AI keeps it a step ahead of regulation. 

In April 2020, ahead of New Zealand’s charter, the Canadian government began requiring impact assessments for all AI systems within the government and in March 2021, US financial regulators requested information from financial institutions on their use of AI, Gavaghan says.

While most countries are trying to tackle AI regulation, he says it’s highly unlikely there will be an overarching global approach to regulating AI. 

“The attitudes around the world are really quite different. China, the US and Europe, all have quite different approaches. I don’t think it’ll be a detailed rule across the world. But I will say that if the European Union implements something like this new law, then New Zealand’s going to have to set up and take notice.”