There is an old statistical adage that, towards the start of this endless ennui* I wrote about. While it has appeared in various formats over the years, the general thrust is the same: “All models are wrong, some models are useful”.
The basic meaning is wonderfully simple — modelling cannot ever capture the ‘truth’, because that is not its purpose. We create models to mathematically codify our predictions, and like all guesses these are prey to our assumptions.
Thus, we don’t categorize models into ‘right’ or ‘wrong’, because it’s a waste of time — they’re all going to be wrong to some degree. We can never fully realize the complexity of human experiences with even the most complex maths, because our inputs are confined to the things we know. Imagine trying to account for every single potential transmission of COVID-19 in a statistical framework, from the casual contact of two people on public transport to the lengthy exposure in a movie theatre. Even the best, most sophisticated models only take the first steps in the tangled web of interconnectivity that we call society.
All models are wrong. But some of them, some few, are useful.
This has been easily recognizable in COVID-19 predictions. Most of the models that were passed around in early March predicted that we’d all either be dead and buried or free from COVID-19 forever by now. And while it was fun to giggle at tech-bros going viral with unfettered certainty that was disproven merely days later, it’s actually worth thinking about the quality of all these predictions and what this means for us moving forward. The next few months are going to be pivotal, as we see the impact of countries reopening, as well as the growing concerns from places that have put little effort into mitigating COVID-19 at all.
The point is that our reliance on predictions is not over. It’s more important than ever that we understand the possibilities that lie before us, because they may very well be the difference between life and death sometime soon.
So, here’s a short list of things to watch out for when you see someone making a prediction about the coronavirus. Some of this may seem obvious, but it’s amazing how many people have been taken in over the last few months by pretty graphs based on total nonsense.
1: Distrust Certainty
If there’s one take-home that I can give you from this whole article, it’s this — distrust anyone who is certain of the future. If a prediction is firm, solid, and confident, it’s probably so wildly off base that you can ignore it entirely. The one common thread among all good epidemic models that I’ve read is uncertainty. No one knows that the future holds, so the best predictions come with caveats aplenty.
2: Check Assumptions
To a great extent, a model is no more than the assumptions it’s based on. I can write hundreds of lines of code, but if I’m assuming that 0.00001% of people will ever catch COVID-19 then no matter how complex my model seems it will be totally worthless. If you read a prediction, check the assumptions behind it. If you can’t find the assumptions, well, that tells you all you need to know about the reliability of the model, doesn’t it?
And once you’ve found the assumptions, check them against reality. Given what we know, is this model reliable? For example, we know that the infection-fatality rate of COVID-19 is likely to be ~0.5–1% — if the model assumes something wildly out of sync with that, it’s worth taking a second look to see if it’s really as reliable as you thought.
3: Look For Trusted Sources
I’m usually not one to recommend acceptance of expertise over factual analysis, but if there’s one thing we should’ve all learned over the last few months it’s that infectious disease modelling is incredibly hard. It is so impossibly difficult to know how and why things change, from the impact of lockdowns to the spread of coronavirus in aged care, that what you really want is someone who has been doing this sort of thing for a long time. It is not actually that difficult to develop a quick-and-easy Susceptible/Infected/Recovered (SIR) model for COVID-19. Don’t be fooled by the calculus — anyone with a few hours on their hands and statistical software can come up with their own fancy-sounding model, which makes it more important than ever to find an expert who knows what they’re doing and listen to them.
4: Check For Updates
The last, and perhaps most important, point is that during the pandemic nothing stays true for more than a week, two at most. The blog I wrote at the end of April? Obsolete by the second week of May. That drug the president was sure worked a few weeks back? Probably worthless.
The only constant is change.
Because of this, what you read today will inevitably be wrong tomorrow. The COVID-19 “worst-case” predictions hit the headlines, but the less terrifying revisions barely made a dent, even though they were far more important. If you want to be informed during this pandemic, you have to keep checking up, otherwise at best you’ll only be seeing part of the picture.
Some Models ARE Useful
Ultimately, the point of this article isn’t to deter you from all predictions — some of the modelling that has been done during this pandemic has been nothing short of incredible. As a colleague told me at the start of all of this, in between school visits for contact-tracing, “modelling has really come a long way. I think it’ll be really helpful for us now”.
We don’t know for sure what the impacts of modelling have been — ironically, we can only model what these might be — but if we can say nothing else it’s that we are generally more informed during COVID-19 than we have been at any time in the past. Some models may have been misleading, and some were clearly hilariously wrong, but others have provided enormous value in the fight against this virus.
We’d be much worse off without the hard work of infectious disease experts around the world.
Which brings me to the end, the final important point. Something poignant, memorable, and easily summed up in a single line so you can share it on social media.
Except, I find that I don’t have one.
“Trust less” isn’t quite right. “Choose your sources carefully” is obvious, but not the whole message. “Stop freaking out about coronavirus predictions” is always a good idea, but given the current situation in the US — where the obvious outcomes of reopening are being ignored in favor of political expediency — there are certainly some things to freak out about. And telling you to keep checking the news is such boringly trite advice that I can only imagine it has entirely lost its impact by now.
Overall, I think the main take-home is the same thing I’ve been saying for months: predicting what’s going to happen next is mind-bogglingly hard. No one really knows. We can say what’s the most likely outcome, and use that to drive behaviour — as good a time as any to remind you to socially distance — but realistically that is only a small part of the story. The main component to all of this is how governments respond, and often that is an even harder thing to predict than the spread of disease.
We might see a resurgence of cases and the dreaded second wave. We might see COVID-19 vary wildly with the changing of the seasons. We could even see government restrictions rolled back with no impact on case numbers — it’s unlikely, but the reality is that there are too many factors to know any of this for sure.
All models are wrong. But in this ever-changing world, the best advice I can give is to try and base your decisions on models that are, at least, useful.
*Gid’s brain: “Has it been weeks? Months? Years? *checks date* Oh right, March 18th. THAT IS NOT LONG ENOUGH AGO.”