The Parliamentary Commissioner for the Environment has sent a shot over the bow of those leading work to fix a tool used to calculate fertiliser run-off
Concerned at slow progress fixing a flawed water quality tool, the Parliamentary Commissioner for the Environment Simon Upton has written to four ministers.
It follows a report he wrote in December 2018 that found Overseer, the main tool many councils use to measure nitrogen run-off, to be seriously flawed.
The letter, published on the commissioner’s website, was sent to Ministers David Parker, Damien O’Connor, James Shaw and Megan Woods.
“… I am concerned that the progress has been slow. I have furthermore, some concerns about the sequence in which work may end up being tackled.”
After the polite scolding, Upton goes on to provide a report card on what has been done and then explains to ministers what needs to happen – and in what order it should be done.
It’s a high-stakes topic in more than one regard. Water quality issues are likely to be an election year issue and there’s also considerable money on the table with $43 million set aside in last year’s budget to improve tools including Overseer.
It appears to be the first time Upton has written a follow-up letter after releasing a report.
The six-page letter does little to inspire confidence the recommendations given in his report are being closely followed.
Overseer – the accidental black box regulatory tool
Overseer is an online software model initially developed by the fertiliser industry to help farmers calculate how much fertiliser to use to maximise pasture growth.
It also estimates likely fertiliser run-off. Over time it has become a regulatory tool several regional councils use to manage pollution in waterways.
It’s now jointly owned by fertiliser companies Ballance and Ravensdown, along with the Ministry for Primary Industries and AgResearch.
These organisations have all poured money into the tool, but fixing it to a point where it could be confidently used as a nationwide-regulatory tool would likely cost millions more.
Speaking to Newsroom in 2018 after producing his report, Upton highlighted the sporadic investment and the government’s ambiguous stance on whether it should, or shouldn’t be, a regulatory tool.
“I’m not saying Overseer is a dog, this is a tool that’s had three decades of investment, some of it impressive and some frankly less so,” he said.
“The pattern of investment has been stop/go, some trials here and there, with the government funding the bulk of it. But even though taxpayers were funding it [the government] was never saying ‘We want this to be used’.”
Upton’s 2018 report found Overseer to be flawed as a regulatory tool and said it was time for hard questions to be asked and answered.
“Overseer does not meet the levels of documentation and transparency that are desirable in a regulatory setting … there is no full, publicly-available, comprehensive description of the scientific framework.”
Upton’s recommendations and what’s happened
Upton has suggested one of the first questions that should be answered if the government wants to use a model to estimate nutrient loss.
If so, an evaluation should be undertaken to assess whether Overseer could be up to the task of being a regulatory tool.
In Upton’s view this would involve assessing numerous capabilities. Only once this is done, does he think work should begin on improvements. Not doing so, he writes in his letter “…would put the cart in front of the horse and could lead to poor use of public funds”.
He also cautions against clinging to a view that Overseer is the answer.
“I believe you need to be open to the possibility that a comprehensive model evaluation may conclude that Overseer is unsuitable for use in a regulatory context, or is only suitable to perform limited tasks in a regulatory setting subject to certain improvements.”
From 10 recommendations given in his original report, it appears four actions are under way with no single recommendation completed.
So far a panel of experts has been assembled for a peer review, but a year on the review has not commenced.
Another issue he raised in his report was that because of the part commercial ownership, its source code is secret and unable to be peer-reviewed by scientists.
This could also mean it would be difficult to use in a regulatory setting. Upton recommended Overseer become open source. His January letter notes the government is seeking advice to help the government determine its position on shifting to open source.
He thinks the question should be ‘how’, not ‘if’, a shift to open source is made.
Overseer is also prone to error. Widely cited reports, which the company confirmed, estimate that nitrogen losses the model predicted were off by 25 to 30 percent. This was a best case scenario as it didn’t take into account errors caused by farmers inputting incorrect values.
Upton notes an uncertainty and sensitivity analysis is under way but commented:
“While this work began almost a year ago, progress appears to be slow, with no information in the public domain to date.”
There’s also an issue of regional variation. With different soil types around the country, the tool would need to be calibrated correctly for the region it’s used in to give accurate results. He notes a reference group has been set up and one technical workshop held.
Upton told Newsroom he is yet to receive a response to his letter, but expects one in due course.
Many critics
Overseer has had its share of critics and has at times been called the ‘Novopay of agriculture’ by farmers. It’s also drawn concern from group of mathematicians who say its approach to crunching the inputs is flawed.
Massey University Professor Emeritus of Industrial Mathematics Graeme Wake is one of those raising red flags for some time.
“It needs evaluation by people who know what they’re doing and suggest alternatives – which are there. They should know that. No one would use Overseer overseas in their right mind.”
In a joint statement that Wake, former IPCC working group director Martin Manning, Massey agricultural senior scientist Tony Pleasants and a retired associate professor of mathematics, John Gamlen, gave to Newsroom last year, the group suggested the mathematical model underlying Overseer be replaced with a model that reflects interactions between different biological processes. They also asked to see inside the black box tool.
The group gave a presentation to the ministries last year, but to date their services have not been requested.
Overseer CEO responds
Overseer CEO Caroline Read said via an emailed statement after the publication of the article above that the PCE report “could not identify any significant flaws in Overseer for what the tool says it can model.”
“What he did identify was that Overseer doesn’t provide the level of real-time specificity that some regulators are seeking,” Read said.
Progress in addressing the recommendations of the PCE took time because it was complex and considerable investment was required, she said, adding the recent appointment of Jacquie Harper as Chief Scientist at Overseer would help develop the tool.
Read said Overseer acknowledged there needed to be more transparency of the model development process.
“We are also working with the Overseer owners on a peer review process and an uncertainty analysis,” she said, adding Overseer’s calibration programme had been underway for some time.
“It is also important to clear up confusion about the tool. OverseerFM is a nutrient budgeting model — not a real-time nitrogen loss calculator.
“OverseerFM uses the information entered about the farm and its management to simulate the fate of nutrients that are brought into or created within the farm. The results produced by Overseer are a representation of the farm system, not the absolute value of a single year’s losses. Farmers and growers can use it to assess how changes in farm management will impact on leaching losses from the farm over time.”
Read said Overseer did have a role to play in compliance, pointing to a report on by Gerard Willis of Enfocus that found there were appropriate ways to use Overseer in regulation.
She said there was no basis for claims Overseer was “seriously flawed.”
“Some areas of the model are more certain than others due to variable data availability. Testing and evaluation was always part of the model development process,” she said.
“Claims of inaccuracy are misleading and often reflect concerns that Overseer isn’t providing the sort of information desired, rather than the modelling is not working as intended. Like any model, OverseerFM has a level of uncertainty in the way it understands the real farm system. There is no way to define an individual margin error on a result in Overseer because that would require specific assessment of the data describing the farm. We are, however, in a process to assess the uncertainty inherent in the modelling.”
(Updated by Bernard Hickey to include comments from Overseer CEO)
