Who punishes whom, when and how? Questions like these are tested in social science studies. | Photo: Martial Trezzini / Keystone

Over seven years, researchers in the social sciences have conducted a project to assess the degree of reproducibility of research findings in their discipline (‘Systematizing Confidence in Open Research and Evidence’, the SCORE Project for short). They investigated over 160 studies to determine their reproducibility, replicability and robustness, and published three articles in the journal Nature explaining what they found.

Reproducibility means obtaining the same results using the same data and methods as the original study; replicability means that repeating the study leads to a similar result; and robustness means that even slightly different methods lead to a similar result as in the study being repeated. Overall, the SCORE team found that only roughly half of the studies investigated fulfilled at least one of these criteria. The behavioural scientist Ilka Gleibs was one of the team working on replicability, and has published an article about their work on a blog of the London School of Economics: “Many commentaries have read these findings as a story of failure”, she says. But she goes on to emphasise the possibility of a different approach: “I want to make the case for a different reading: one in which the glass is half full, and the SCORE project gives us good reason to be optimistic about the state of social science”.

“Uncertainty is inherent to scientific knowledge. It is not a defect to be eliminated but a feature to work with”.Ilka Gleibs

Gleibs also reviewed the Nature article on reproducibility: “Uncertainty is inherent to scientific knowledge”, she says. “It is not a defect to be eliminated but a feature to work with”. In her opinion, the study in question opens up a perspective for a future based on transparency: “a path with better policies on data and code sharing, and more open methods”.

Gleibs is also insistent that the credibility of research results is a multifaceted thing that isn’t just about “replication, reproduction and robustness”, but also encompasses aspects such as “theory, knowledge about contextual changes, controlling for bias and having valid methods”. She believes that the SCORE project has served to open up a new discussion about a fundamental issue: Just how should we be assessing the credibility of studies in the social sciences?

CC BY-NC-ND