Reports from early trials of the CovidCard shows the technology underperformed in complex environments with lots of metal, dust or people, like cafes and parties, Marc Daalder reports
A mid-May trial of the CovidCard contact tracing technology showed the device tracked fewer than two-thirds of clinically significant contacts when used in a cafe.
Documents publicly released by the Department of Internal Affairs after an Official Information Act request by Newsroom include the results of two trials of the CovidCard. While the technology appears to work well in certain environments, like an office space, and regularly logged more contacts than people could remember on their own, it struggled in more complex scenarios like a cafe, a party or a construction site.
Despite these results, the Government decided by early August that the technology worked to the degree needed. A Government-run trial that is scheduled to take place in Rotorua would only test how the CovidCard integrated into New Zealand’s contact tracing system and whether people would voluntarily wear it, Government Digital Services Minister Kris Faafoi told Newsroom.
“We’ve been convinced that the technology in terms of the card works. The cards talk to each other – we know that. It’s a different thing around how the card integrates into the contact tracing system, around privacy things that we need to keep in mind and, the most important thing is, are people going to use it?” he said at the time.
However, he clarified in a statement to Newsroom that the Rotorua trial would have a broader scope.
“The Rotorua trial will test people’s take-up and willingness to use the Covid Card. That’s why we decided on the Rotorua trial,” he said.
“Rotorua also provides an important further step to confirm whether or not the card will function as it needs to in real life settings so the Government can decide if it is a useful addition to New Zealand’s contact tracing processes.”
Andrew Chen, an expert in technology and society and a research fellow at the University of Auckland’s Koi Tū – the Centre for Informed Futures, said the released documents showed the technology needed more trials to prove its efficacy.
“I wouldn’t go so far as to say it doesn’t work. But given that the previous statements that were publicly made claimed it could record clinically significant contacts with greater than 90 percent, I think this data suggests that that is true only in nice, clean, controlled circumstances,” he said.
“The results in this report would indicate that under noisier conditions, you might be looking at closer to 30 percent error rates, depending on your definition of error. As a very rough rule of thumb, that is probably similar to what you might expect from the [smartphone-based] Apple/Google Exposure Notification Framework in similar environments.”
The first trial, in Nelson Hospital in early May, found the CovidCard detected six times more contacts than people remembered, on average. It also found contacts were generally 16 minutes longer than participants remembered.
However, the trial didn’t seek to ascertain whether the card was logging contacts accurately. The second trial, in the Waikato on May 15, simulated scenarios in a cafe, a construction site, a taxi, two parties and two offices. Each set of interactions was recorded by CCTV cameras to establish “ground truth” for how many clinically significant contacts actually occurred. That means someone within two metres of another person for 15 minutes or longer.
In the end, the card fared best in office scenarios, where more than 90 percent of contacts were recorded and only a small percentage of the contacts recorded were false positives. In more complex scenarios, it struggled. Some of this was environmental – Bluetooth signals can experience interference in places with large amounts of metal (like a cafe) or dust (like a construction site). Some of it had to do with crowds in close quarters, as with a party.
The cards in the cafe on average recorded only 63 percent of the contacts they should have. At the construction site and the two simulated parties, the cards had a higher recall rate but more than half of the contacts recorded were false positives – that is, they recorded contacts with cards that weren’t actually within two metres for 15 minutes.
Dean Armstrong, the technical lead for the private sector team behind the CovidCard project, told Newsroom he wasn’t concerned by the results of the Waikato trial.
“Those results weren’t worrying, and I don’t think should be worrying,” he said.
“You want to have some objective measures but at the end of the day, you’re always going to have scenarios that a digital contact tracing solution won’t be able to get right. So it’s a question of whether if gets you above a baseline. In the absence of anything else, the baseline is human recollection.”
Armstrong said the cards only return raw data about length of contact. While an algorithm processes the raw data into epidemiological information and delineates what amount of contact should be logged as a contact of concern, the data itself can be used to inform contact tracing as well.
“What you’re faced with is the difference between how you report hard accuracy of a card when you draw a line and how it’s actually used in practice. The right way to use a system like this is to have clinician-informed risk assessment. Manual contact tracing process is always going to be at the heart of contact tracing.”
In other words, contact tracers can look at the data and use it to remind people about contacts they may have had during investigatory interviews. They can then make their own judgment as to the risk each contact represents. So if the CovidCard showed someone had contact with others for 20 minutes on a given day and that was actually a scenario where the infected person was wearing a mask and sitting two rows in front of others on a bus, the contact tracers could deem that a lower transmission risk than a face-to-face conversation in a restaurant.
Armstrong also said that the performance of the card could be improved in the future because the team had developed custom hardware – the trials were run on off-the-shelf hardware from overseas companies – and because the algorithm can easily be edited.
“Definitely that can be improved and even improved over time. The things that turn raw reports off the card into this prioritised list of contacts, that stuff happens on the server side.”
This addresses one of the concerns of critics of the CovidCard, who worried that embedded firmware on the cards couldn’t be edited if the parameters needed to be changed as more information about the virus becomes available. While some firmware will still be difficult to change, the algorithms processing the data are run server-side, allowing them to be tweaked with ease.
Chen noted that the documents only cover work up until early July, prior to the new hardware developed by the CovidCard team. But he says more trials and more development work would have been needed before a nationwide rollout.
“Overall, my reading of this is that more trials were needed and that there was probably room for more development and iterative improvement on the current card design,” he said.
That’s different from the sort of work the Rotorua trial is expected to investigate. Armstrong said the primary focus of the Rotorua trial will be to determine whether people will wear CovidCards – likely on a lanyard around their necks – without being required to.
“The questions that I believe Government most need to answer, the important question is, can we get people to wear it? There’s no point in proceeding if you don’t think people will wear it.”
But he still thinks that further development can and should occur alongside the trial, so the cards will be able to roll out more quickly if the experiment in Rotorua returns positive results.
“If there’s going to be any significant rollout of this, then there’s absolutely a process that we need to go through or somebody needs to go through to make sure this is a polished product. If your motivation is to save every cent, then you do the minimum needed at the moment, which is to go and answer that question – will people wear it? And then while that’s done, you go and you do more development. You just put the development on hold while you answer that question,” he said.
“If your motivation is that if we’re going to need it and if it’s going to work, then we need it fast, then you progress a bunch of other things in parallel. While you’re answering that question, you forge on, you do the product development. But Government has decided they want to run the trial. That’s what the fixation is on.”
A spokesperson for the Ministry of Health said the trial would “investigate how the cards would work in a real-world scenario, whether they are compatible with our contact tracing systems, and whether the public would accept and use them”.
“Also part of the programme was a technology trial to ensure the cards provide the robust privacy and security controls needed to carry out a community trial, as well as other technical evaluations to prove the efficacy of the technology for any wider use. During this process, it had become clear that the prototype cards do not include adequate controls. Further development and supply options have been considered and work is moving quickly to resolve these issues.”
The spokesperson said a decision on any rollout would be made by the end of the year.
