This year, there have been two massive outbreaks. The first, and most obvious, is the global pandemic of COVID-19 that is changing our world even as you read these words. The other, that has been commented on in hilarious fashion, is the massive surge in armchair epidemiologists, with seemingly every person in the world suddenly an expert in how to analyze public health research.
Now, there’s no argument that this is a bit weird. It’s obviously not that easy to become an expert in infectious disease epidemiology, which makes the fact that we’re all suddenly debating everything from reproductive rates to airborne transmission pretty amazing.
But while it may be a bit silly, it’s also totally understandable. We live in an age where all of human knowledge is at our fingertips, so becoming a bit knowledgeable about the things that are drastically impacting our lives has never been easier than it is now.
Trouble is, it’s easy to get these things wrong. The ideas aren’t always simple, and often they are fiendishly counter-intuitive. None more so than sensitivity, specificity, and the wonderful complexity of testing people to see if they have a disease.
Sensitivity and specificity are concepts that relate to the accuracy of diagnostic tests. In other words, they are incredibly important when it comes to knowing if someone does or doesn’t have COVID-19. Depending on the sensitivity and/or specificity of a diagnostic test, your antibody or swab result may mean quite a bit or nothing at all.
Pretty important stuff, that.
Let’s kick off with some quick definitions.
Sensitivity = the proportion of tested people who are correctly identified as being sick (i.e. false negative rate).
Specificity = the proportion of tested people who are correctly identified as NOT being sick (i.e. false positive rate).
The basic issue with sensitivity and specificity, and testing in general, can quite easily be summed up with a simple (age-old) example:
Say 1 in 1,000 people have a disease. You have a test that correctly identifies when someone is sick 100% of the time (i.e. no false negatives). It also correctly identifies when someone ISN’T sick 95% of the time (i.e. 5% false positives). What is the likelihood that someone testing positive for the disease actually has it?
The simple, intuitive answer, that many doctors themselves give, is that your test will be right 95% of the time. Some people hedge a bit and argue that actually it’s more like 50%, or as low as 20%. Weirdly, all of these answers are wrong. The correct answer is that, in this scenario, a positive result will only be right in 2% of cases.
What’s going on here?
Well, the maths is pretty simple. Let’s say you test 1,000 people. You correctly catch the 1 person who has the disease (remember, no false negatives). But then, you test the remaining 999 people and get 50 positives, even though none of them have the disease (5% specificity). So you have 50 false positives for every true positive, giving you a positive predictive value (i.e. how likely it is that someone who tests positive actually has the disease) of just 2%.
The thing is, your prior probability changes everything. If only 1 in 1,000 people have a disease, then even a moderately low specificity makes your test worthless. Even if the specificity in the above example had been 99.9%, your test still would’ve only been right half of the time. If you look instead at a population where 100 in 1,000 people have the disease, most of your positive tests are right even at 95% specificity.
As I said — totally counter-intuitive, but incredibly important*.
What does it mean for coronavirus and your life?
Sensitivity and specificity are important in different ways depending on your test. For example, if you are running a test to see is someone is currently infected with COVID-19, your biggest worry is false negatives — you really don’t want to send a sick (and probably infectious) person home without treatment, but you can always retest a positive result if they don’t feel sick.
On the other hand, if you what you’re really interested in is how many people in the population have been infected ever, and you’re testing their antibodies, you will be much more concerned about false positives. Missing a few people who had the disease won’t change your numbers much, but unless most people have been infected even a very low false positive rate is an enormous issue.
So, for example, if you look at the recent testing that’s been going on in Wuhan, you can pretty quickly see an issue. In the city of China where the coronavirus pandemic was first picked up, they are testing literally everybody to see if anyone is currently infected with the disease. The test they’re using — a form of PCR — can have a very high false negative rate, particularly in the early stages of infection. What this means is that even though they’re testing every single person in the city, they might only catch a fraction of people who are currently infected with — and potentially transmitting — COVID-19.
On the other end of the scale, specificity is a big problem for some recently released antibody studies. If you test 1,000 people, and get 2 positives, but your false positive rate is 0.2%, then it’s possible that both of these positives are false. If your estimate of the number of people infected with COVID-19 could be entirely explained by the false positive rate of the test, then the results are pretty hard to interpret. This has huge implications for things like the fatality rate of the disease, because if you overestimate how many people have had COVID-19 then you will underestimate how fatal the disease is.
What Does This Mean To You?
All of this is fascinating, but population statistics rarely mean very much to the individual. What we all want to know is what our test results mean — if you are positive for coronavirus antibodies, does that mean you can go outside again?
In general, the problem with all these stats is that they aren’t very useful unless you know everything about the situation. The likelihood that your individual test result is trustworthy is pretty dependent on both the accuracy of the test and the number of people who’ve been sick near you. If you’ve never been sick, and no one you know has, even if the test is very accurate a positive result is a bit meaningless. On the other hand, if your entire family has been down for weeks with COVID-19 symptoms, and everyone else tests positive, a negative result for you is pretty likely wrong.
In other words, it’s really hard to give you a single answer. It all depends.
And this is why most public health authorities aren’t giving free license to roam for people who have positive antibody tests. Similarly, the same set of issues make coronavirus immunity passports unlikely to be a good solution to our travel dilemma.
Ultimately, the reality is that coronavirus testing is complex. Positive and negative results have to be interpreted in context, which is why most of the testing is being done by doctors and public health experts.
The best advice I can give you is to ask your doctor what your COVID-19 test means, and listen to their advice.
You can also listen to me host the fantastic Sensationalist Science podcast here:
*As an added bonus, you’ve just learned the basis of Bayesian statistics, and it only took half of a blog post. Well done you.