Antibody Tests Evaluated

How did the Abbott and Roche SARS-CoV-2 antibody tests stack up under independent evaluation by Public Health England?

Dr Alasdair Scott, MBBS MRCS PhD

May 30th, 2020

Abbott and Roche are two large medical device companies, based in the US and Switzerland respectively. They both released test platforms to measure blood levels of IgG antibodies against the SARS-CoV-2 virus which have recently been approved for use by Public Health England (PHE). The diagnostic performance of the tests as reported by both Abbott and Roche was excellent but we’ve had to wait a little while to read PHE’s independent evaluation. Here’s how they stacked up.

How did PHE evaluate the antibody tests?

PHE tested the Roche and Abbott antibody test platforms on blood samples from two groups of people. A group of “cases” who had had symptoms of COVID-19 and had tested positive for SARS-CoV-2 with the PCR swab test and a group of “controls”. The “control” blood samples included samples collected from people before mid-2019, prior to the COVID-19 outbreak, which would be expected to contain antibodies to other seasonal coronaviruses but not SARS-CoV-2. The “control” groups also contained samples that were known to be positive for other common viruses such as EBV or herpes virus.

If the SARS-CoV-2 antibody tests from Roche and Abbott work like they’re supposed to, then we would expect the “case” blood samples to return a positive result and the “control” samples to come back negative. With this in mind, we can calculate two main measures of how “good” each test is: the sensitivity and specificity.

The sensitivity is the proportion of people or samples who have had SARS-CoV-2 (“cases”) and get a positive result from the antibody test. For example, if 100 samples were taken from people who’d had SARS-CoV-2 and the antibody test came back positive in 90 of them, the sensitivity would be 90/100 or 90%. We want sensitivity to be as high as possible to minimise the risk of a “false-negative”, which is where someone has had the virus but incorrectly gets a negative antibody test result.

In contrast, the specificity is the proportion of people or samples who have not had SARS-CoV-2 (“controls”) and get a negative* result from the antibody test. For example, if 1000 samples were taken from people who had not had SARS-CoV-2 and the antibody test came back negative in 990 of them, the specificity would be 990/1000 or 99%. Again, we want the specificity to be as high as possible to minimise the risk of a “false-positive” result, which is where someone who has not had the virus incorrectly gets a positive antibody test result. This could be particularly dangerous for CARS-CoV-2 because a false-positive result might make someone think they have some protection from the virus when they actually don’t.

How did Abbott do?

For the evaluation of the Abbott test, PHE had 96 “case” samples and 757 “control” samples.

Sensitivity

  • The overall sensitivity was 92.7% (95% CI 85.6-97.0).
  • The sensitivity for samples collected at least 14 days after the onset of symptoms was 93.4% (95% CI 85.3-97.8).

This means that if we tested 1000 people who had had coronavirus, 934 would get a positive result but 66 people would incorrectly get a negative result (a false-negative).

Specificity

  • The overall specificity was 100% (95% CI 97.79-100).

This means that if we tested 1000 people who had not had coronavirus, they would all get a negative result.

And how about Roche?

For the evaluation of the Roche test, PHE had 93 “case” samples and 472 “control” samples.

Sensitivity

  • The overall sensitivity was 83.9% (95% CI 74.8-90.7).
  • The sensitivity for samples collected at least 14 days after the onset of symptoms was 87.0% (95% CI 77.4-93.6) .

Specificity

  • The overall specificity was 100% (95% CI 99.1-100).

What can we take away from this?

Firstly, both the Roche and Abbott SARS-CoV-2 IgG test have a very high specificity (around 100%). This means that there’s a very low chance of getting a “false-positive” result.

Secondly, as we would expect, both tests work better when there has been at least two weeks between the onset of symptoms and the sample being taken for testing. This is because it takes time for your body to produce the antibody that these tests measure.

Thirdly, the Abbott test may be slightly better (more sensitive) than the Roche test for detecting previous SARS-CoV-2 infection. The sensitivity of the Abbott test for samples taken at least 14 days after the onset of symptoms was 93.4% versus 87.0% for Roche. However, we can’t say for sure that one test was better than the other because, a) this was not a head-to-head comparison, b) the 95% confidence intervals for the two senstivity results overlap and, c) the numbers of samples tested was quite small.

What do these results mean for me?

Sensitivity and specificity are all very well and good but what do they actually mean for you and your test result? To understand this we calculate two more important values - the positive predictive value (PPV) and the negative predicitive value (NPV).

The PPV is the chance that you’ve had the virus if you get a positive antibody test result. The NPV is the chance that you haven’t had the virus if you get a negative result. These values depend somewhat on how common infection with the virus is in population in the first place. The higher the proportion of people who have been infected with the virus, the more likely it is that a positive antibody test result is correct and a negative test result is incorrect. The opposite is also true.

Currently, researchers estimate that around 10% of people in metropolitan areas like London have been infected with coronavirus. Outside of cities, the level is probably closer to 5%. Of course, certain professions such as health or social care workers may have higher levels of infection.

If we assume that 10% of the population has had coronavirus, then if you take the Abbott test:

  • The positive predictive value is 100%
  • The negative predictive value is 99.3% (95% CI 98.5-99.8)

If we assume that 5% of the population has had coronavirus, then if you take the Abbott test:

  • The positive predictive value is 100%
  • The negative predictive value is 99.68% (95% CI 98.3-99.9)

In summary, if you get a positive SARS-CoV-2 antibody result with the Abbott platform then these data suggest that it’s pretty much certain that you’ve had the virus. Similarly, if you get a negative result then it’s very unlikely that you’ve had the virus.

Lastly, it’s important to understand that we’re constantly getting new data on these tests and the disease itself. This is a very new virus and we have a lot to learn. Although it’s likely that previous infection with SARS-CoV-2 will provide a degree of immunity we don’t know for how long or how strong. Regardless of your antibody test result, do not assume you are immune and be sure to follow the current advice and guidance from PHE.

More info