Do Lockdowns Work?

Almost a month into 2021, and despite the promise of a fabulous new year the reality has already set in that the next 12 months probably won’t be as much of a breeze as we’d hoped. Vaccine programs are slowly rolling out, but given the scope of them it’s unlikely that we’ll be back to anything resembling normal any time soon. Facing rising case counts, and overwhelmed hospitals, many places around the world have decided to go into a second or even third lockdown to try and quash their truly out of control epidemics before the situation gets even more dire.

It’s been a bit of a depressing start to the year, to be sure.

In this wild maelstrom of clashing information, a story went viral last week claiming that all of this was unnecessary. you see, according to Newsweek and some other publications, a study had shown that there was no benefit to lockdowns over other voluntary measures that could be taken by governments.

Unfortunately for all of us, this research doesn’t really show very much. The reality is that it is much harder to tell if lockdowns — or any other measures against COVID-19 — work than you’d imagine.

The Study

The paper itself was a new piece of modelling research that looked at case numbers and government interventions between 10 countries total at the start of the pandemic. The scientists took the confirmed case numbers in 2 “less restrictive” countries — Sweden and South Korea — and compared them to the case numbers in 8 other “more restrictive” countries including Iran, the US, UK, and a number of other places.

Overall, the scientists failed to find a benefit for the implementation of “more restrictive” interventions — the so-called lockdowns, mostly consisting of stay-at-home orders and the like— although they did find a benefit for government intervention overall. This lead them to conclude that it might be possible to achieve similar reductions in case numbers in many places using less restrictive interventions rather than full-on lockdowns.

This sounds like good news — we can get the same impact without the pain! — until you look more closely at the research. There are some issues that make it hard to take any meaning away from this study at all.

Laws and Lockdowns

The first thing to note is that this study is not examining a new idea. The marginal benefit — the benefit of enacting an intervention on top of the things you’ve already implemented — of, say, stay-at-home orders has long been questioned. It is not unlikely that, if you’ve already closed most businesses, schools, and the like, that forcing people to stay inside will have a much smaller impact than it would if you hadn’t taken those measures already*.

But the problem with studies on this topic — and the new research in particular — is that answering this question is incredibly difficult. Firstly, what is a “lockdown”? In some places, like NSW Australia, the “lockdown” period had much less tight restrictions on going outside than in many areas of the world. Did we have “more restrictive” interventions (as defined in this research), or did the fact that we had a long list of reasons to go outside (including meeting up with a friend for exercise) make this “less restrictive”?

South Korea is another great example of this. While the country did not implement laws preventing people from moving outside, depending on your measure of “restrictive” you could easily argue that they were pretty harsh. For example, there aren’t many places where the government can collect mobile phone, credit card, and other data from anyone infected to track their movements and trace contacts without consent if necessary. South Korea also has had one the longest in-person school closures of any country in the world, with many students spending 9+ of the last 12 months learning from home.

On top of this, the interventions themselves are very different. Sweden closed schools for in-person teaching, but did it very differently to South Korea and other countries. Indeed, it’s hard to find two places where the laws and regulations for COVID-19 are exactly alike, which makes sense when you consider how different the culture is in many of these places.

This problem compounds when you consider that the new research only looked at a tiny sample of 8+2 countries. There are dozens of other “more” and “less” restrictive countries in the world for which there is data available, but instead of looking at these we only have a small fraction of the world represented.

It’s also incredibly difficult (if not impossible) to determine causality using these research methods. The authors basically ran some simple models looking at cases before and after each intervention was put in place in each country, which means that there are innumerable confounding factors that may be at play here. It’s possible that cultural and economic differences between the “more” and “less” restrictive countries caused the divergent case counts, or that any one of an almost uncountable number of factors were behind these disparities. Given the lack of any effort to control for this in the research, it’s impossible for us to know.

The study also didn’t include any lag for implementation. This is important, because most countries — especially early in 2020 — implemented their restrictions at a time when cases were already rising steeply. Given testing and reporting lags, particularly in some of the slower to update nations (such as Sweden), you would not expect official confirmed case counts to decrease until days or even weeks after policies were implemented even if they were working from day 1. Think about it this way — most people take 4–5 days to show symptoms after being infected, and then delay a day or two before being tested, add another day (or more) for the results to come back, and immediately we’ve got a 7–10 day lag between officially reported case numbers and actual infections. This means that if you don’t at least try and control for this lag in your model, you end up with a largely meaningless result.

Another weakness that the authors pointed out in their discussion is that they used official case counts. As I and co-authors showed in our paper on COVID-19 IFR, and many others have done before and since, official case numbers for COVID-19 are usually wrong, often by a very large amount. Particularly in early March 2020, they were often more a reflection on how many tests a country was running, rather than a true indication of how many people had the disease in each place.

This means that the differences observed between countries could simply be down to how many tests each place was running at the time and have nothing at all to do with the effectiveness of the government interventions. As with many of these issues, we simply don’t know if official cases are a good proxy for true cases, and it makes this study (and other similar ones) quite hard to interpret.

Overall, the study has so many holes that it’s hard to take much or any meaning away from it. It’s certainly possible that “less” restrictive interventions were the key to controlling COVID-19, but it’s also possible that the results were totally meaningless and have no relevance whatsoever for COVID-19 at all.

What Does This All Mean?

The bottom line is that, as with many things in epidemiology, it is really hard to answer the question of whether lockdowns work, or to what extent. We know that in theory limiting in-person contact will drive down the number of people who catch COVID-19 in an area, and various interventions clearly work to do that, but putting a specific number on the marginal benefit of each intervention is incredibly hard, particularly when you consider them in context. Closing in-person dining might be of huge benefit if it’s the only thing you do, or add only a small fraction to the total reduction if it’s part of a larger package of laws.

The new paper adds almost nothing to that pool of evidence except to show yet again that answering the question of whether lockdowns work is really very hard.

What we do know, from both this new paper and others, is that government interventions definitely can and do reduce COVID-19 cases. While there might be a debate about which interventions are best, and which have the lowest cost for the greatest benefit, we do know that these interventions can control the epidemic as they have done in countries around the world.

What we definitely don’t know yet — and probably won’t for some time to come — is which interventions work best in which location. When it comes to the question of whether restrictive government interventions work, the reality is that we can easily say that they probably prevent COVID-19 infections, but precisely how many infections per intervention is a far harder question to get at.

So do lockdowns work?

As with many things, the answer is fiendishly complex.

If you enjoyed, follow me on Medium, Twitter or Facebook!

*Note: This is not to say that so-called “lockdown” interventions are useless, or do not contribute meaningfully, but if you have already reduced the reproductive number from 3 to 1.1, the additional benefit of going from 1.1 to 0.9 may be harder to discern, but in many ways just as vital in terms of stopping the epidemic from spreading.