A recent project spearheaded by Brian Nosek, a founder of the nonprofit Center for Open Science, invited researchers to analyze the same data around a prompt: Do soccer referees give more red cards to dark-skinned players than light-skinned ones? Twenty-nine teams, 61 expert analysts, and wide variety of methods were used.
The results? 20 teams concluded the answer was yes (with widely varying magnitudes), 9 teams found no significant relationship.
The variability in results wasn’t due to fraud or sloppy work. Even the most skilled researchers must make subjective choices that have a huge impact on the result they find.
All is not lost, however. These disparate results don’t mean that studies can’t inch us toward truth. For instance, it is hard to look at the results and say there’s no bias against dark-skinned players.
The important lesson here is that a single analysis is not sufficient to find a definitive answer. Every scientific result is a temporary truth, one that’s subject to change when someone else comes along to build, test and analyze anew.
But, if this is the case, how do we make sense of news clippings that claim A causes B? More on that next week.
Science isn’t broken. It’s just a hell of a lot harder than we give it credit for. – Christie Aschwanden
Source and thanks to: The FiveThirtyEight Blog – Science isn’t broken