When Every Story Is A Success Story

How only seeing survivors skews our perception of the truth

Image for post
Image for post
Turns out searching for ‘survivor’ on stock photo sites gives you lots of results for islands. Source: Pexels

There’s a common problem that a lot of people have when they start engaging with other humans online, particularly on Twitter. You start reading up on a topic that you think is a fairly fringe belief — say, that the Earth is flat — and instead of finding only one or two absurd people advocating for it, you somehow find hundreds. Even thousands.

If you don’t believe me, check out the #FlatEarth hashtag on twitter. The mind boggles

And it’s not just faux-intellectual buffoonery that seems disproportionately visible. Everything from dodgy diets to weird supplements seems to have dozens of success stories and few, if any, failures.

Want to find out if becoming a pure carnivore is right for you? It seems as if there are hundreds of people who it’s worked for and virtually none who’ve failed. Wondering what the best supplement combination is for your hair? Somehow, you’ll find thousands of testimonials online about how successful kelp is, and very few saying that it’s bogus.

Image for post
Image for post
Pictured: Basically magic, apparently. Source: Pexels

The problem is that, when we look at the science, it often disagrees with these positive messages. The most obvious example is the Flat Earth theory, which was disproven more than 2,000 years ago and has been ridiculed by virtually everyone since then. Diets are, of course, harder, but generally speaking there’s no reason to believe that any one diet has a huge advantage over another in terms of weight loss (although some might have small advantages for specific conditions like diabetes). Kelp might help with your hair — although it hasn’t really been studied and doesn’t seem that likely — but it can also be a source of elevated arsenic levels which means it’s probably not that great an idea either.

So why does everyone seem to hold these unlikely beliefs? The answer may lie in the people we are looking at.

Maybe, we are only seeing the survivors.

Image for post
Image for post
Pictured: Survivors? Source: Pexels

When you run medical experiments, you are always trying to control as many variables as you can, so that you can be as sure as possible that you have an equal comparison between the thing you are interested in and your control group. If you are testing whether cars are faster than bicycles, for example, and don’t control for road type, you might find that the bikes do better simply because the tests were conducted on a mountain bike track and the cars all crashed.

We call these things, the elements of a study that influence the outcome but are out of our control, sources of bias. And as you can imagine, there are a lot of them. The Cochrane Collaboration — an internationally renowned research group considered the gold standard for rating studies — identifies dozens of potential sources of bias in its handbook.

One major type of bias that we talk about is called survivorship or survival bias. This type of bias crops up when we select people who are already survivors for our intervention group, and then compare them to average people in our control. Because the people who we’ve selected are already survivors, they look better on paper than the people in the control group, and our intervention looks much better than it is.

Image for post
Image for post
“The tribe has spoken” Source: Pexels

Research into drugs gives us a great example. Let’s say that you want to know whether statin therapy — a type of drug that lowers cholesterol — prevents people from having a repeat heart attack. You take a group of people who have had heart attacks, divide them into those who have filled a statin prescription and those who haven’t, and check which group has more heart attacks.

The result? Magic! Statins appear to prevent about 40% of heart attacks. For such a cheap and simple intervention — an off-patent drug that is relatively safe — you’ve stopped dozens of heart attacks and many deaths.

But wait. There’s a problem here. You see, the group we’ve selected are the people who filled their prescriptions. The other group are people who didn’t manage to fill their prescription. Now, that might be because they just didn’t get to the pharmacy, but some of the things that can prevent people from filling prescriptions include being in hospital or being dead.

Image for post
Image for post
Hard to get to a pharmacy outside of hospital if you never left in the first place Source: Pexels

Because of the way that we selected our groups, we are only seeing survivors.

It turns out that, if you compare the groups more equally, the benefit from statins drops a lot. A group of scientists actually did this study, and they found that depending on how you selected your treatment group — the people receiving statins — the benefit of the drugs ranged from a 40% reduction in heart attacks to no reduction at all.

That’s a big difference.

How you select your sample group makes a huge difference in what you see as a result. What does this have to do with fringe beliefs online?

Well, there’s a good chance that we’re only seeing survivors.

The thing is, for someone to advocate for something online, they almost have to be a survivor. Very few people go on Instagram or Facebook to tell us about all of the diets that have failed them, but there are literally thousands of stories of amazing success.

If you look online, you’ll almost exclusively end up with a positive answer, even if the reality is entirely different. Take Black Salve, for example. If you search on Twitter, you’ll see hundreds of people promoting it as a cure for skin cancer, because it’s worked for them. This might lead you to believe that it’s effective.

In fact, the opposite is the case.

Image for post
Image for post
I would’ve put a picture of a black salve wound here, but it’s seriously disturbing so here’s a kitten instead. If you’re super keen on seeing something gross, click here. Source: Sereja Ris

Black salve is an escharotic, which basically means it kills all the cells it comes into contact with, leaving behind a nasty black scar. Using it for skin cancer is a bit like pouring acid on your skin every day — it’s potentially possible that it might kill the cancer, although in most cases it doesn’t, but even then you’ll be left with a massive hole in your body and be in an enormous amount of pain. There are innumerable cases of people who took black salve instead of conventional therapy and died because of it, but you don’t hear those because they are not the survivors.

Survivorship bias isn’t the only reason that there often seem to be a million voices shouting about something ridiculous, but it is a big one. It doesn’t matter if 99.9% of people are convinced by the strong and abundant evidence that human activity is causing the globe to warm, if 0.1% are not, and they are the only people who yell about it, those are the only voices that you will hear.

So next time it seems like the internet is crowded with illogical ignorance, or you see a million people recommending something that you know doesn’t work, remember that often it’s not the world that’s going crazy.

It’s just that you’re only seeing the survivors.

If you enjoyed, follow me on Medium, Twitter or Facebook!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store