Stephen Downes

A study published in Nature last week - behind a paywall, naturally - confirms what many previous studies have shown: about half of all studies in the social sciences cannot be replicated. The situation is slightly better for education research. "When the same data is analyzed differently by various researchers, the conclusions drawn can vary significantly...  The choice of analysis method has a significant impact on the findings. Decisions regarding data cleaning, variables, and statistical models are crucial and can lead to different outcomes." I would apply these findings not only to individual studies, but also systemic reviews. "These results do not call into question the credibility of earlier research," says Balázs Aczél of Eötvös Loránd University. "Rather, they show that presenting a single analysis often does not reflect the true extent of empirical uncertainty. Ignoring analytical variability can lead to a false sense of confidence in scientific conclusions." See also: Science, Stewart-Williams, National Tribune, Kau.se, Phys.org.