Why science is hard – Part II – The 200 words project

(Continued from Part I)

Why do we see headlines about coffee being good for us one day and bad the next? It is possible that the original study was flawed- career incentives around paper publication have created bogus journals. But, it is gradually becoming harder to fake it on the internet with online forums and comments on all journal websites. Self-correction is a critical part of science and retractions are on the rise.

science is hard

The main issue is in such studies is that isolating how coffee affects health requires lots of studies and lots of evidence, and only over the course of many, many studies does the evidence start to narrow to a conclusion that’s defensible. The variation in findings is not a threat – it just means that scientists are working on a hard problem.

This uncertainty doesn’t mean that we can’t use findings to make important decisions. We should make the best decisions we can with the current evidence but take care not to lose sight of its strength and degree of certainty AND stay open to new data. It’s no accident that every good paper includes the phrase “more study is needed” — there is always more to learn.
(More on how failure is actually moving science forward in the last piece of this series next week..)

Science is great, but it’s low-yield. Most experiments fail. That doesn’t mean the challenge isn’t worth it, but we can’t expect every dollar to turn a positive result. Most of the things you try don’t work out — that’s just the nature of the process. Rather than merely avoiding failure, we need to court truth. – FiveThirtyEight


Source and thanks to: The FiveThirtyEight Blog – Science isn’t broken

Why science is hard – Part I – The 200 words project

A recent project spearheaded by Brian Nosek, a founder of the nonprofit Center for Open Science, invited researchers to analyze the same data around a prompt: Do soccer referees give more red cards to dark-skinned players than light-skinned ones? Twenty-nine teams, 61 expert analysts, and wide variety of methods were used.

The results? 20 teams concluded the answer was yes (with widely varying magnitudes), 9 teams found no significant relationship.

science is hard

The variability in results wasn’t due to fraud or sloppy work. Even the most skilled researchers must make subjective choices that have a huge impact on the result they find.

All is not lost, however. These disparate results don’t mean that studies can’t inch us toward truth. For instance, it is hard to look at the results and say there’s no bias against dark-skinned players.

The important lesson here is that a single analysis is not sufficient to find a definitive answer. Every scientific result is a temporary truth, one that’s subject to change when someone else comes along to build, test and analyze anew.

But, if this is the case, how do we make sense of news clippings that claim A causes B? More on that next week.

Science isn’t broken. It’s just a hell of a lot harder than we give it credit for. – Christie Aschwanden


Source and thanks to: The FiveThirtyEight Blog – Science isn’t broken