Introduction

Harald Atmanspacher and Sabine Maasen

Reproducibility has become a hot topic both within science and at the interface of science and society. Within science, reproducibility is threatened, among other things, by new tools, technologies, and big data. At the interface of science and society, the media are particularly concerned with phenomena that question good scientific practice. As bad news sell, today problems of reproducibility seem to be ranked right next to fraud. The economy, and especially the biotechnology economy, is interested in innovation based upon novel yet robust knowledge and politics in the so-called knowledge societies seek to base their decisions on best evidence, yet is regularly confronted with competing expertise.

A key step toward increasing attention to deep problems with reproducible findings in science was the paper "Why most published research findings are false" by Ioannidis (2005). One among many recent urging proclamations following it was published in The Scientist magazine (Grant 2012):

The gold standard for science is reproducibility. Ideally, research results are only worthy of attention, publication, and citation if independent researchers can reproduce them using a particular study's methods and materials. But for much of the scientific literature, results are not reproducible at all. The reasons and remedies for this state of affairs was the topic of a recent panel discussion titled "Sense and Reproducibility", held at the annual meeting of the American Society for Cell Biology in San Francisco, California. . The panel offered suggestions, such as raising journals' publication standards, establishing the use of electronic lab notebooks at research facilities, and helping laboratory supervisors provide improved supervision by reducing the size of labs.

Since about a decade voices abound - both in academia and in the media - that lament lacking reproducibility of scientific results and urgently call for better practice. Given that scientific achievements ultimately rest upon an effective division of labor, it is of paramount importance that we can trust in each other's findings. In principle, they should be reproducible - as a matter of course; however, we often simply rely on the evidence as published and proceed from there. What is more, current publication practices systematically discourage replication, for it is novelty that is associated with prestige. Consequently, the career image of scientists involved with cutting-edge research typically does not include a strong focus on the problems of reproducing previous results.

And, clearly, there are problems. In areas as diverse as social psychology (Nosek 2012), biomedical sciences (Huang and Gottardo 2013), computational sciences (Peng 2011), or environmental studies (Santer et al. 2011), 1 serious flaws in reproducing published results have been and keep being detected. Initiatives have been launched to counter what is regarded as dramatically undermining scientific credibility. Whether due to simple error, misrepresented data, or sheer fraud, irreproducibility corrupts both intra-academic interaction based on truth and the science-society link based on the trustworthiness of scientific evidence.

Among the initiatives introduced to improve the current state of affairs we find workshops, roundtables, and special issues addressing the topic, e.g., in Nature (Schooler 2011, Baker et al. 2012). The journal Biostatistics changed its policy with a focus on reproducible results in an editorial by Peng (2009). The journal Science devoted a special issue to the topic in December 2011, and later revised its publication guidelines concerning the issue of reproducibility (McNutt 2014). Three prominent psychology journals jointly established a "reproducibility project" recently, 2 and the journal PLOS ONE launched a "reproducibility initiative" in 2012. The European Journal of Personality published recommendations for reproducible research (Asendorpf et al. 2013) as the result of an expert meeting on "reducing non-replicable findings in personality research."

Funding agencies have also joined forces: e.g., the National Science Foundation of the United States created the "Sustainable Digital Data Preservation and Access Network Partners" (DataNet) program to provide an infrastructure for data-driven research in 2007. And, very recently, the National Academy of Sciences of the United States hosted an internal symposium titled "Protecting the Integrity of Science" (Alberts et al. 2015). An extensive report on reproducible research as an ethical issue is due to Thompson and Burnett (2012).

In sum, these examples point to an increased attention toward reproducibility as a topic sui generis. They testify to an increasing interest in reproducibility as a scientific ethos that needs to be upheld - even more so, as new tools and technologies, massive amounts of "big data," inter- and transdisciplinary efforts and projects, and the complexity of research questions genuinely challenge and complicate the conduction of reproducible research. Many of them call for methods, techniques (including their epistemic and ontological underpinnings), and/or best practices that are intended to improve reproducibility and safeguard against irreproducibility.

The challenges of sound reproducible research have moved into the focus of interest in an increasing number of fields. This handbook is the first comprehensive collection of articles concerning the most significant aspects of the principles and problems, the practices and prospects of achieving reproducible results in contemporary research across disciplines. The areas concerned range from natural sciences and computational sciences to life sciences and social sciences, philosophy, and science studies.

Accordingly, the handbook consists of six parts. Each of them will be introduced by separate remarks concerning the background and context of aspects and issues specific to it. These introductory remarks will also contain brief summaries of the chapters in it and highlight particularly interesting or challenging features.

Part I covers contextual background that illuminates the roots of the concept of reproducibility in the philosophy of science and of technology (Tetens, Zimmerli), and addresses pertinent historical and sociological traces of how reproducibility came to be practiced (Steinle, Collins). Part II frames the indispensable role that statistics and probability theory play in order to assess and secure reproducibility. Basic statistical concepts (Stahel), new ideas on model selection and comparison (Shiffrin and Chandramouli), the difficult methodology of meta-analysis (Ehm), and the novel area of data mining and knowledge discovery in big-data science (Estivill-Castro) are covered.

Parts III-V are devoted to three main areas of contemporary science: physical sciences, life sciences, and social sciences. Part III includes the viewpoints of computational physics (Bailey, Borwein, and Stodden), severe novel problems with reproducibility in complex systems (Atmanspacher and Demmel), the field of extreme and rare events (Kantz), and reproducibility in climate research (Feulner). Part IV moves to the life sciences, with articles on drug discovery and development (Martic-Kehl and Schubiger, Folkers and Baier), the neurobiological study of cortical networks (Lengler and Steger), cognitive neuroscience (Anderson) and social neuroscience (Vogeley).

Part V offers material from the social sciences: a critical look at the reduction of complex processes to numbers that statistics seems to render unavoidable (Porter), innovative strategies to explore question order effects in surveys and polls (Wang and Busemeyer), original views on public opinion research (Keller), issues of reproducibility as indicated in the "blogosphere" (Reinhart), and an in-depth study of notorious problems with reproducibility in scholarly communication (Franzen).

Part VI widens the perspective from reproducibility as a problem in scientific disciplines (in the narrow sense) to literature and literature studies (Bezzola Lambert) and psychopathology and psychoanalysis (Emrich). There is a clear shift in viewpoint here from the attempt to repeat experiments and reproduce their results to an analysis of why strict repetition is not only impossible but also undesirable. Another article (Atmanspacher) proposes that the way reproducibility is studied needs to be adapted to the granularity of the description of the system considered. The final contribution (Maasen) leads us back to the science-society link and the impact of extrascientific forces on research that often remains underrepresented or even disregarded.

This volume investigates the principles, problems, and practices that are connected with the concept of reproducibility, but there is a fourth "p-word" in addition: prospects. In some of the chapters the point is not only to understand principles, address problems, or scrutinize practices - there can also be a strongly constructive dimension in research questions that, on the surface, suffer from a lack of reproducibility. One pertinent example in this volume is the paper by Anderson, who builds on the limited reproducibility of neural correlates of mental states and proposes new ways of interpreting them coherently. Another example is the radical shift in theories of decision making...