The Director of Loyola University Medical Center’s clinical microbiology laboratory is reported as saying that rapid flu tests are a public health risk. Here’s some of what he said and then my explanation as to why it is misleading or just plain wrong:
Rapid influenza diagnostic tests used in doctors’ offices, hospitals and medical laboratories to detect H1N1 are virtually useless and could pose a significant danger to public health, according to a Loyola University Medical Center researcher.
“At Loyola, we determined four years ago that the rapid tests for influenza detected only 50 percent of the patients who were positive,” said Paul Schreckenberger, Ph.D., director of Loyola’s clinical microbiology laboratory. “I can flip a coin and get the same results as I could with those tests. So what’s the value of the tests? I can flip a coin for free.” (Medical-News)
We’ve discussed this before, but since the underlying concepts seem poorly understood (even by clinical laboratory directors), it’s time to discuss it again.
First, let’s review how we evaluate diagnostic tests. There are two dimensions: reliability and accuracy. These terms have technical definitions in epidemiology. Reliability is what most people call repeatability. If you perform the same test twice on the same specimen, will you get the same result? Extremely reliable tests may also be extremely inaccurate. If you measure blood pressure with a broken device so that it gives the same (wrong) value each time, the device is extremely reliable (in the technical sense) but very inaccurate. Accuracy is a measure of how close you come to the true value. With blood pressure that can be measured quantitatively and inaccuracies can be of various degrees, but for qualitative measures like “flu or no-flu” it is either right or wrong. There are two kinds of ways to be wrong: you can say someone has the flu when they don’t, or that they don’t have the flu when they do. There are also two kinds of ways to be right: if the person has the flu your test can correctly say they do; and if they don’t have the flu, your test can correctly say they don’t.
It is the latter two measures that are used to express accuracy of a test, because the former two, although convertible to the latter, are measuring error, not accuracy. These two measures (how good is the test in picking up flu and how good is it in picking up non-flu) are called sensitivity and specificity. It is the relatively poor to moderate sensitivity of the rapid antigen test (50%) that Dr. Schreckenberger is complaining about and likening it to a “flip of a coin.” Unfortunately that is not the right way to think about 50% sensitivity because it ignores two other vital pieces of information, the specificity of the test (an independent measure of accuracy) and the prevalence of flu in the community (a feature independent of the test’s accuracy but important for evaluating its use).
Note that the test is not being applied to everyone in the population, only those who have influenza-like illness (ILI). There are many causes of ILI besides influenza and there are more kinds of influenza than swine flu, but it turns out that just about the only type and strain of influenza currently circulating in the community is swine flu, so that if you have influenza at this time it is almost certainly swine flu. Thus the relevant question is not whether you have a positive test if you have swine flu (that’s sensitivity and the 50% figure Dr. Schreckenberger quotes), but whether you have swine flu if you have a positive test. That’s something he doesn’t discuss, and it’s not clear he understands the difference because he likens the outcome of the test to the flip of a coin. But it isn’t. Given 50% sensitivity, if someone comes in that I know has swine flu, I could flip a coin and get the same result as the rapid test. But I don’t know if they have swine flu or not. If I did I wouldn’t have to do any test. I only know that they have an ILI and I want to know if that ILI is swine flu. In mathematical terms I am conditioning on a positive test, not conditioning on having swine flu, but you can get an understanding of this technical point with some examples.
Suppose it’s a non-pandemic year and summertime, when most ILIs are caused by viruses other than influenza virus. Let’s say that out of every 100 ILIs, only 6% are caused by a flu virus. That’s 6 people. The rapid test will pick up 50% of them (that’s what 50% sensitivity means), or 3 actual cases. Three of the 6 will go undetected by this test. But there may be others with positive tests who don’t have swine flu. That’s a reflection of a lack of specificity. For the rapid test specificity is pretty high, often 90% – 100%. That sounds pretty good, but in the instance where flu isn’t common among ILIs, as in a non-pandemic summertime, you will see what happens. There are 94 people out of 100 in the example who don’t have flu. If the specificity is 100%, then there are no false positives and the predictive value of the test is 100%: everyone with a positive test also had the flu (even though 3 were missed). If the specificity is 90%, 90% of 94 is about 10 cases (rounding up), so of 13 positive tests (3 real cases and 10 positive tests that were other kinds of viruses), the chance of being a true case if you had a positive test is only 23%, much worse than Schreckenberger’s “flip of a coin.”
Now consider what happens if the proportion of flu among the ILIs is 30% or 50%, something that can easily happen in a pandemic or just a regular flu season when a lot of the ILIs are being caused by flu. For 30% flu among the ILIs, 15 are being picked up by the rapid test, while of the 70 ILIs that are not flu, either 100% (79) or 90% (63) are negative by the test. That means that either 0 or 7 positive tests are wrong. In the first case, every positive rapid test will be correct, while in the second 15 of 22 (15 + 7) will be correct, or 68%. If 50% of ILIs are flu, the situation is worse for the Schreckenberger claim: a positive test will correctly identify swine flu in 50 out of 55 positive tests (for 90% specificity), or 91% of the time.
So much for the claim that a 50% sensitivity means the test is no better than a toss of a coin. That would be true only if you used it in a circumstance where you already knew the answer (that the person had flu already). But it’s useful to also consider the flip side. What should you conclude if the test says you don’t have flu? With a 6% prevalence of flu among the ILIs, 90% specificity means that 84 of the 94 people without flu will have negative tests and 3 of the 6 who do have flu will have negative tests. That means that a negative rapid test for flu will be correct on average 84/87 = 96.6% of the time (for 100% specificity it is 97%), while for the worst case, 50% prevalence, a negative test will be right 64% of the time, still better than the flip of the coin Schreckenberger alleges.
What’s the lesson here? When there’s hardly any flu around, a positive rapid test isn’t very good at predicting whether you have flu, but a negative test almost certainly means you don’t. When there’s a lot of flu around (half of ILI being flu), a positive test can be quite accurate (depending on the specificity; with 100% specificity it is 100% correct, but even for 90% specificity it is correct over 90% of the time). A negative test will be correct only about 2/3 of the time, but better than a flip of the coin. With an intermediate case of 30% ILI being flu (quite conceivable during the flu season; remember this isn’t 30% of the population, but 30% of those with flu-like symptoms), a positive test will be right more than 2/3 of the time and a negative test will be right 81% of the time, both better than a “flip of the coin” (these are both for 90% specificity; with greater than 90% specificity the test will do better).
CDC has been quite clear that clinicians should not rely on negative tests to say a patient doesn’t have flu, since in current circumstances that could be wrong about a third of the time. That’s not a flip of the coin but it’s still a lot of missed cases. But with a lot of flu around and virtually all of it swine flu, a positive test is a very strong indication you have swine flu. Not a flip of the coin.
The point of the post, though, is not to advocate for rapid tests (their value depends on any particular test’s sensitivity and specificity and the prevalence of flu in the population, and all three of which vary), but to discuss some aspects of testing that seem to be poorly understood. Even by clinical laboratory directors.