ANALYSIS: In mid-2014, Stats NZ, the national statistics office, produced a detailed business case for the next census.
The census provides crucial information that underpins government policy, including drawing electoral boundaries. But costs were going up and the Government had mandated a 5 percent saving over the next two surveys.
The estimated budget for the 2018 census was $119 million. In May 2014, a month before the business case was submitted, an external quantitative risk analysis (QRA) by Australian consultancy Broadleaf suggested more money was needed – and more people. It estimated a further $6 million might have to be spent on the census, about half of that on personnel.
Stats NZ thought it knew better. Instead of asking the Government for more money, it trimmed $3 million from its overheads and tried to absorb other costs within its baseline budget. The business case concluded: “Whilst recognising the risks identified by the QRA, Statistics NZ remains confident it can successfully deliver the modernised 2018 census within the Budget 2014 appropriation of $118.9m and is not seeking additional funding.”
This misguided hope, a ‘Yes Minister’ attitude to delivering a high-risk operation with significant changes, set the tone for a haphazard census, costing almost $127 million, that has now claimed the job of Chief Statistician Liz MacPherson.
An independent review, released yesterday, catalogues numerous errors across the operation.
The picture is chaotic. IT systems delivered late, an inadequate field workforce, not enough paper packs, slow delivery of forms to households. The “lessons learned” section could read like a first-year university management course, with obvious lines like “Keep things simple”, “Testing is critical”, and “Plan for the worst and execute for success”.
Despite Stats NZ being forced to curtail the expected information released from the census, and having to advise the Government of its looming financial hole, it has somehow managed to pull a rabbit out of the numerical hat. By waving a statistical wand, using what’s called administrative data from other Government sources, some census statistics will, ironically, be of better quality than the 2013 census.
It has allowed some people – including Minister James Shaw – to make glossy statements about how well the problem’s been fixed. But the review of the disastrous census is so damning that, perhaps, MacPherson’s head isn’t the only one that will roll.
Internal storms
Nature seemed to conspire against the census.
The November 2016 Kaikōura quakes damaged Stats NZ’s Wellington headquarters, knocking out its IT systems for weeks, and leaving some systems unstable for months. The following year’s dress rehearsal was cut short by a Civil Defence emergency in Whanganui.
But Stats NZ was also its own worst enemy. It could have deferred the 2018 census after the Kaikōura quakes, but carried on. It asked for $5 million more from the Government and got $2 million. It didn’t push for more. The 2017 test in Whanganui didn’t use the IT systems used in census field operations.
The review report says: “Due to the lack of testing, Statistics NZ did not appear to fully understand how the various components of the model needed to work together to achieve success.”
Independent reviewers Murray Jack, a director and management consultant, and Connie Graziadei, a former assitant chief statistician at Statistics Canada, say census staff focused too much on online risks, perhaps because of a major incident during Australian census night in 2016. This distracted Stats NZ from fully thinking through the risks of other aspects of its operation, including what it should do if responses were low.
Ministerial briefings leading up to census day were largely focused on possible technological failures, rather than field operations or coming up with contingencies for a low response. The review says: “We find a level of optimism in the reporting to ministers that was not always consistent with the level of issues being managed by the programme.”
That might give Minister Shaw an excuse, as he doesn’t appear to have been adequately briefed. But what of senior census managers?
The report notes the Government Statistician – who has now resigned – is “accountable” for the census, but the responsibility for day-to-day leadership and management was delegated to the census general manager. Until May last year that was Denise McGregor’s job.
She sits on the Stats NZ executive leadership team, as a deputy government statistician and deputy chief executive. Does she have questions to answer? And considering the census’s IT delays and problems, including a faulty address list, what responsibility lies with chief digital officer Chris Buxton?
Earnest but needs to listen more
The design of the 2018 census was “feasible” and the budget “sufficient”, the review says. Indeed, it believes the model should be retained, despite the poor result.
Despite a change to digital-first, in which people mailed an internet access code were encouraged to complete the census online, the objectives were “not unreasonable”.
(On the plus side, the census’s internet collection system was “secure, stable and easy to use”, the review says. More than 80 percent of forms were completed online, smashing the expected target of 70 percent.)
But few issues and decisions were referred to the census’s governance boards.
“There was a sense that the census programme felt it could manage the programme and did not formally engage with the governance boards in a structured way,” the review report says.
As it was, reports by census executives to the various boards were “overly positive”, and the seriousness of issues were underplayed.
“Feedback from a number of external stakeholders describes a culture in Statistics NZ that is earnest and committed, but one that needs to take a more user-centric perspective, listen more, and be more transparent.”
Known problem made worse
It was only in August 2017 that Stats NZ decided how many paper forms to print. By then, NZ Post wasn’t able to print enough.
Paper packs weren’t given to field workers making initial visits to hard-to-reach areas, with no internet or mail service, where response rates are generally low. Then the understaffed field team was told to drop letters at the doorstep or in the mailbox rather than knock on doors, to ensure they could get round more properties before census day on March 6.
People living in isolated areas were forced to ring up and request paper packs. Entire communities missed out on them, including many Māori and Pasifika families, who had been identified in the previous census as being undercounted. (Of 1.3 million printed “visit packs”, in English and bilingual, only 530,000 were delivered.)
Māori response rates were 68.2 percent, down from 88.5 percent in 2013. Pasifika dropped to 65.1 percent from 88.3 percent. And 15-29-year-olds, as a sub-group, dropped from 88.5 percent in 2013 to 75 percent last year.
Concerns raised but dismissed
Stats NZ decided to mail-out census invitations to 80 percent of dwellings instead of 70 percent. As a result it reduced the field staff “target” from 3000 – about 40 percent of the 2013 census workforce – to 2300.
The review says the drop to 3000 was “too aggressive”, and the further reduction was based on a mathematical model “without careful consideration of respondent behaviour”. There’s no evidence of the decision being raised with governance boards or “that options or risks were considered”.
Even the lower staff target wasn’t reached. After census day, Stats NZ needed about 1500 field workers to knock on doors of dwellings that hadn’t responded. But its peak workforce only reached 900. Field operations were “severely hampered by insufficient staff”.
Even when the system worked it sometimes wasn’t believed. The census’s management information system identified that individual response rates were tracking below an acceptable “tolerance level”. But because of a lack of testing and scant knowledge of the system, there was a “lack of confidence in the information”. Issues were raised by those monitoring the system but “appropriate action wasn’t taken”.
Concerns were also raised about letter drops being favoured over contact with field staff, and the decision to provide those staff with fewer paper packs. “These concerns did not appear to influence decisions.”
“Stay the course. The paper will come.” – An example of post-census day optimism within Stats NZ
Here’s where culture kicks in.
“The overall impression is of a programme very focused on delivery but not as open to advice as was desirable,” the report says. “There was a sense of ‘don’t worry, this is hard, but everything is going well and we will deliver the census as planned’.”
Another section of the report notes an “overly optimistic view” – “Stay the course. The paper will come.” – in the face of concerningly low responses from individuals. As a consequence, management was slow to react.
This blind optimism also found its way into a “Readiness for Service” report, finished in June last year, three months after census day. The report said that notwithstanding the “miss” of low responses, it had still exceeded performance expectations “in all areas whereby Stats NZ had direct control”. This should be “shouted from the rooftops” as an “outstanding achievement in the delivery of New Zealand’s first predominantly digital census”.
This comment surprised reviewers, who noted it was well-known by then the response rate was below 90 percent, and work was underway to plug the gaps. The statement “reflects a level of optimism that was not supported by facts”.
Change is on the way
The language in Jack and Graziadei’s review makes clear what they think didn’t happen last year.
Leadership of the census programme should have “experience in managing large statistical programmes, strategic leadership, and the ability to foster an open and collaborative culture”. The programme plan should identify “a critical path that enables effective management of dependencies and risks and provides for appropriate contingency planning”. And timelines should be adjusted “to allow sufficient time for testing and a full dress rehearsal”.
Some action has already been taken, and other recommendations have been made.
An independent body should advise the Government Statistician on matters like ethics, privacy and security, the reviewers say. A survey design authority, chaired by the Chief Methodologist, should be established. A single census governance board should be created – and be consulted on all major decisions.
In some parts of the country, it might be necessary for census workers to fill out the forms directly, like in the past. Meanwhile, the reviewers back a move to re-establish an audit and risk committee.
The report says the increased use of administrative data worked and should happen again – although it’s noted there’s a job to do to build public acceptance.
In summary, more checks, more balances, management more prone to listening, and an organisation more able to respond quickly to problems. There might also be more field workers brandishing more paper forms.
To ensure mistakes aren’t repeated, there might also have to be more accountability.
