Temperatures are rising in Parliament’s corridors over last year’s census failure.
The fact one-in-10 people didn’t participate in the five-yearly national survey has already delayed the release of data, as officials try to fill the gaps, and prompted an external review. Now, Government Statistician Liz MacPherson, threatened with contempt by a parliamentary select committee if she didn’t respond, has revealed that five percent of responses were only partially completed.
That means more than one-in-seven, or about 700,000 people, didn’t fully complete the bungled $126 million survey.
The National Party is calling for the next census, due in 2023, to be brought forward. It’s concerned about the re-drawing of electorate boundaries and that the population numbers underpinning decisions worth tens of billions of dollars aren’t reliable enough. Its attack dog, Nelson MP Nick Smith, said the “cock up” will “create problems for years”, Stuff reported. Economist Brian Easton told the news website the response gap was so large it could make the data useless for research.
Statistics Minister James Shaw told journalists yesterday that Stats NZ is taking its time in releasing data so it can get “as good, if not better quality information from Census 2018 as we have from any other census in the past”. He added: “Dr Smith is way out of line and he needs to back off.”
However, a university lecturer also questions the reliability of the census data and suggests the undercount may have implications for public funding. Will the Minister also accuse him of scaremongering?
“You just don’t know because you haven’t measured.” – Richard Arnold
“It is disastrous,” Richard Arnold, a lecturer at Victoria University of Wellington’s school of mathematics and statistics, says of the census undercount. In the past, it’s been in the order of 2 percent or less – making this census “absolutely not up to spec”.
It will still produce publishable information, he says, but they’ll need to be interpreted with “great caution” and certain output goals won’t be met.
“The problem is that whenever you have undercount you’ve got some risk of error; risk of bias. But as long as the undercount is really held low – even if it varies across age groups, even if it varies across ethnicities – its effect is minimal.
“It isn’t a problem in that case. But when it gets out to this kind of level, it’s a really big problem.”
Large groups that might have had significant numbers of people missed include people aged in their teens and early 20s, Māori and Pacific Islanders, and the elderly, he thinks. The bias introduced by a large undercount can be “very real”, Arnold says.
“And they’re just unknowable. You just don’t know because you haven’t measured.”
Not that the census was useless. Arnold says it depends on the purpose for which the data is used.
“If you, for example, wanted to know whether the prevalence of smoking has gone down or up, you can still find this. Unless you think that smokers are less likely to have access to technology, those kinds of things – they’ll still be there.
“Global social trends will still be visible, but they may be slightly less reliable.”
A minor win was the high number of people responding online, he says. And there may be, in coarse terms, things that can be deduced from the data. The estimate of the number of people “missed” will have come from Stats NZ’s post-enumeration survey, Arnold says. So just add 700,000 to your total responses and – pow! – you’ve got your population estimate.
“The problem is you don’t know where those people are. So that has impacts for regional government funding allocation – in health services and education allocation – which are regionally stratified.
“You don’t know how old they are or how young they are, so that affects your ability to plan for building schools or plan for aged care. There are just all of those complexities, that mean they [have to] make assumptions and make guesses.”
Using old census data
The mere delay in the release of census population data – before the debate over their accuracy – is already having an effect on population-based public funding.
Education Ministry advice, released to Newsroom under the Official Information Act, shows a lack of census data has delayed a recalculation of school deciles and updating a so-called “isolation index”, which affects schools’ funding. Reviews will now be based on 2013 data.
A briefing to Health Minister David Clark said new population projections had changed funding for district health boards. “Note that these are based on Census 2013 data as Census 2018 data is not yet available.”
Meanwhile, Minister for Māori Development and Local Government Nanaia Mahuta says she received no direct briefings about the potential effects of the census undercount, between last July and March 11.
Victoria University of Wellington’s Arnold worked for Stats NZ in the early 2000s, on its household labour force and immigration surveys, often using census data. He says what distinguishes a census from a sample survey is it “gets in everywhere”. Small groups, like migrants and refugees, will be worried about being “missed” by the census and having a reduced profile, he says.
“It’s interesting to see how they are growing and changing over time. There’ll be people from minority religions who want to see how many they are and how they’re changing.
There are little groups everywhere. Even regional areas, where people want to know how the rural population is changing.”
Politicians are rightly jumping up and down about the census, Arnold says, and asking whether the $126m budget was well-spent. He says an online survey makes sense – “but it just can’t be done cack-handedly”.
The primary goal of a census is to draw electoral boundaries, he says. “And you’ve had to say there’s some doubt about that – it didn’t meet its primary goal.”
MacPherson will hold a press conference on April 29 to announce when the first census data will be released.
In her letter to governance and administration select committee chair Brett Hudson, MacPherson says: “Stats NZ has increased the use of administrative data to fill as many of the gaps as possible. We have been able to add more detail for most of the five percent of partial responses, and we’re now in the process of evaluating and checking the data to determine quality ahead of release.”
Audit New Zealand will help Stats NZ to assess its new methodologies and approaches. An independent expert review of the census is expected to report back in July.
Arnold doesn’t know the reasons for the large census undercount, but he knows what needs to happen in the next census: “They’ve got to get the response rate up. So, whatever that takes.”
