A new report by the Parliamentary Commissioner for the Environment raises serious questions about a tool many councils are using to try to curb farm run-off. Fixing the system will likely cost millions, reports Eloise Gibson.
A major report has found that the main tool regional councils are using to measure nitrogen run-off to lakes and rivers is seriously flawed.
The online model Overseer is jointly-owned by two fertiliser companies, Ballance and Ravensdown, along with the Ministry for Primary Industries and a Crown-owned science company, AgResearch.
It was originally designed as a commercial tool to help farmers make money by maximising the milk or meat they gain from using artificial fertilisers. It is still used for that.
But because Overseer also estimates how much fertiliser is wasted from paddocks as run-off, it has morphed into a regulatory tool that several regional councils use to manage nitrogen pollution in some of our most troubled waterways. Faced with pressure to clean up waterways, several regions turned to using the only tool they felt they had available for estimating run-off – Overseer.
A new report by the Parliamentary Commissioner for the Environment raises serious questions about the tool’s transparency and accuracy.
Because Overseer is a commercial enterprise, its source code is secret, making it difficult for outside scientists to properly peer-review its accuracy.
Even an internal analysis of how accurate it is, was kept out of the public domain.
A detailed report on the tool by Parliamentary commissioner Simon Upton cites widely quoted reports that nitrogen losses produced by the model may differ from real losses by about 25-30 percent. The report notes that these reviews were never made public but the findings were confirmed by the company. That 25-30 percent variance is the best-case scenario, because it doesn’t include inaccuracies caused by measurement errors, for example if a farmer doesn’t know how much fertiliser or supplementary cattle feed they have used.
The model uses these numbers to make its estimations – for example, it will model how much nitrogen is lost in cattle urine by looking at how much a typical animal needs to eat, how much nitrogen is in their food, and how much gets converted to milk or meat. It isn’t designed to show what happens to the nitrogen once it joins with other substances in a river or lake; it can only work out how much gets lost to the soil beyond the grass’ root zone.
Compounding the potential for wrong results is that most of the soil information and rainfall data that was used to build the model came from two regions, Waikato and Southland. That means estimates for farms with different soil or rainfall profiles might be out by more than 50 percent. Upton’s report says the model might not perform as well in places like Marlborough, Tasman, West Coast, Gisborne, Taranaki and Northland. “In Canterbury, Overseer estimates of nitrogen leaching from dairy farms on light and poorly-drained soils could be anywhere from nearly 40 percent below to 60 percent above the actual leaching rate,” says the report. While Upton notes that no computer model can ever be a perfect match for reality, those are big variations.
On top of the in-built issues, the system is open to gaming by farmers if they know how to manipulate their inputs to get a lower run-off result, the report into Overseer found. “In discussions with farmers, farm consultants and regional councils, numerous anecdotes of deliberate manipulation were reported, as were instances of workarounds, and interpretation differences,” it says. “Where Overseer is used to set a farm nitrogen loss limit …or determine compliance with nitrogen limits, there are significant incentives for the deliberate manipulation of Overseer modelling results.” To tighten compliance he recommends more independent auditing, to see if records are realistic for the farm and to check that Overseer inputs match other data, such as tax records and fertiliser invoices.
Questions over accuracy may create as many problems for farmers as they do for our waterways. While the tool might be accurate enough for making commercial decisions about fertiliser purchases, that is a different matter to figuring out whether farms are breaking the law. Right now, the report says it is hard for farmers to have confidence that the estimates councils were using to make compliance decisions were right. And, at any time, a software update could change Overseer’s algorithms and put a farm in breach of pollution limits – even if the farmer hadn’t changed anything they were doing on the farm, Upton noted. Not helping matters was that councils couldn’t reassure farmers about the inner workings of a tool they didn’t fully understand themselves.
Despite these major short-comings, six regional councils are using Overseer to regulate nitrogen run-off: Bay of Plenty, Canterbury, Hawke’s Bay, Horizons, Otago and Waikato. These councils can require farms to use the tool to estimate nitrogen losses and, sometimes, demonstrate how they plan to reduce impacts far in the future. Another three councils, including Southland, require farms to provide nitrogen leaching reports using Overseer but don’t use the tool to impose any limits on run-off.
Upton’s report says the councils with the most pressing water quality issues have had little choice but to use the tool. Farmers oppose councils using blunter controls to reduce run-off, such as capping stock numbers, limiting imported feed, or restricting fertiliser quantities, because those tools give them fewer options for choosing how to meet water goals. “The severity of the nitrogen problem they face has led them to Overseer. Council staff acknowledge the tool is far from perfect, but blunter tools would be required if Overseer was not available,” the report says.
As for the future, Upton wants the government to decide – quickly – whether it wants councils to keep using Overseer or to find another solution. If Overseer is going to be used, it needs to be transformed into an official, open-source tool, he says. “It is time to open up Overseer. If the Government wants to see the model being used as a regulatory tool then a large measure of transparency is needed,” he said in a statement accompanying report.
The report itself carries on with the open-ness theme: “Overseer does not meet the levels of documentation and transparency that are desirable in a regulatory setting … there is no full, publicly available, comprehensive description of the scientific framework.”
The first task for over-hauling the tool is a major review by MPI, Upton says. Then the Government should tell regional councils whether, and, if so, how, it wants them to use it.
Fixing the tool, if that’s what the government chooses, will likely cost millions.
The report concludes that Overseer’s shortcomings are a result of its origins and the way it’s been funded (or under-funded) over the years. Making it fit-for-purpose would not only require more government cash, but, perhaps, full government ownership, it says.
Upton raises the question of whether it would be appropriate for an open-source tool for monitoring pollution to be owned by two major manufacturers of fertiliser. Yet Ravensdown and Ballance have helped develop and fund the tool, spending close to a million on it in 2017 (roughly the same amount as was spent by the Ministry for Primaries Industries). Over the years, the tool has cost well over ten million, and absorbed significant donated expertise from AgResearch. None of the owners has extracted any profits, other than the goodwill it has generated by building relationships with farmers.
“If the answer (to fertiliser company ownership) is no, then it would be an option for the government to buy [their] ownership stake in the intellectual property,” says Upton.
It’s now up to the Government to decide what it should do, and spend.
