The goal of any research assessment is to evaluate the value or quality of the research in comparison to other research. As quality is highly subjective and difficult to measure, citations are used as a proxy. Citations are an important part of scholarly communication and a significant component of research evaluation, with the assumption being that highly cited work has influenced the work of many other researchers and hence it is more valuable. Recently we have seen new online data sources being researched for this purpose and disruptive ideas with the power to change research assessment, and perhaps even science as a whole, have been born. Altmetrics is the new research area that investigates the potential of these new data source as indicators of the impact that research has made on the scientific community and beyond, and thus possibly also as indicators of the societal impact of research. This book will present some of these new data sources, findings from earlier altmetrics research, and the disruptive ideas that may radically change scholarly communication.
- Presents some of the key ideas and innovations in earlier research that have been driving the evolution from bibliometrics to webometrics, and with the advent of social media to altmetrics
- Discusses the shortcomings and pitfalls of bibliometrics in research evaluation and the potential of altmetrics to overcome some of these shortcomings
- Presents some of the most important data sources of altmetrics, the aggregators, and the different stakeholders
- Reviews current research about altmetrics and discusses possible future trends
- Presents a way to measure and aggregate altmetrics according to the level of impact or type of impact they represent
The first part of the book will discuss the past of altmetrics, its origins, its scientific roots, and its connection with bibliometrics and webometrics. In many aspects the past of altmetrics is also the past of bibliometrics and webometrics, but it needs to be emphasized that the beginning of altmetrics does not mean the end of bibliometrics or webometrics. The three research areas are developing side by side, learning from each other, complementing each other. This part of the book will give an overview of scholarly communication and the research methods involved in "counting, measuring, and weighing" it, namely bibliometrics and more recently, after the advent of the web, webometrics to analyze scholarly communication on the web. The shortcomings and pitfalls of bibliometrics in research evaluation will be discussed and the current standards and practices for most reliable bibliometric analyses will be presented. With that, the technical developments and societal changes that paved the way for altmetrics will be presented. This part will end by focusing on developments in social media, which, as an increasingly important place for scholarly communication, has made altmetrics possible.
Goodhart's law: When a measure becomes the target, it ceases to be a good measure(Goodhart, 1975)
From increasing competition of funding comes the need to find relevant and reliable metrics to help funders decide the most deserving targets for increasingly limited finances. Different aspects of scholars' and researchers' work are being assessed to find researchers with the most potential to make an impact on science and society as a whole. Faculty members' work usually consists of three tasks or three different areas of activities: (1) research, (2) teaching, and (3) public outreach. Productivity in research can be measured from the number of research publications, but in order to assess the quality of that research some other means are required. Teaching activities could be assessed, for instance, from the number of courses, tutoring hours, and student feedback, and activities related to public outreach could be measured from the number of public talks given or appearances in news media. Nevertheless, usually only research activities matter, or at least, matter most.
Some researchers focus full heartedly on research, while others may have more passion for teaching or public outreach. The expectations on research productivity may even vary from one discipline to the other (Dewett & DeNisi, 2004). Yet when evaluating researchers' productivity and performance, most emphasis is usually placed on research activities. This of course depends to some extent on the purpose of the evaluation, but in general, even candidates for an academic position with mostly teaching duties are in the Humboldtian university system mainly evaluated on their publications. Evaluation of researchers has traditionally focused very tightly around the formal research publications, which have of course benefited more productive researchers, at the cost of those scholars that have focused more on teaching or public outreach. This usually means that bad teachers with great publication records are chosen over great teachers with bad publication records.
As the number of researchers and the number of their research products have grown significantly in the last decades, it is increasingly difficult to assess the work of researchers, yet alone the work of research groups, universities, or disciplines, by peer review alone. In peer review the work of a researcher or researchers is evaluated by their peers, preferably experts in the same area of research, who familiarize themselves with the research products of those being evaluated and then give their assessment, depending on the purpose of the evaluation. While this might be possible to do when assessing a couple of candidates for a faculty position, it is not possible to do at any larger scale. It would simply take too long and be too costly to perform. The peer review therefore needs help from methods that can assess large quantities of research products quickly. Enter bibliometrics, the research area that investigates literature using quantitative methods.
In line with the increasing need for quantitative methods for research assessment, interest towards bibliometrics has also increased. The number of scientific publications about bibliometrics has increased almost 100-fold in the last 20 years, from only 97 publications indexed by the Web of Science in 1994, to a staggering 8731 publications in 2014 (Figure 1.1). This alone demonstrates the immense increase in the interest towards bibliometrics, and, with that, research assessment in recent years. Figure 1.1
The increase of publications about bibliometrics as indexed by Web of Science between 1994 and 2014 (data retrieved from Web of Science on February 9, 2015).
Numerous bibliometric indicators reveal to us a story about the desire to quantitatively assess different aspects of research work and scholarly communication, aspects that go beyond productivity alone. More recently that story includes new data sources and disruptive ideas that may change scholarly communication and research assessment for good. These ideas are the result of some technical developments and earlier research about science in itself and about scholarly communication as the backbone of scientific development.
2 Scholarly Communication
Robert K. Merton wrote in 1942 that "the institutional goal of science is the extension of certified knowledge." A single researcher's formal contributions to the extension of this certified knowledge or common knowledge are his or her scientific publications and in every discipline the formal scientific publications are of utmost importance. It is also through publications that the researcher positions himself or herself in the fields of science and it is through the publications that the researcher claims intellectual ownership of his or her ideas, of his or her contributions to the extension of knowledge. These contributions can, however, also be of an informal nature, such as conversations during coffee breaks at conferences or more recently, for instance, on Twitter or other social networking sites. They can also take other forms, such as datasets and software. Both the formal and informal contributions to the common knowledge are part of scholarly communication, but they also define and position the researcher in relation and in comparison to other researchers.
Scholarly communication is the process that starts with a research idea that may be acquired from reading the work of other researchers and that certainly builds upon the work of others. This is followed by the research work and writing of the manuscript, and ends with a formal scientific publication that through peer review is accepted as an "extension of certified knowledge" (Figure 2.1), or, in fact, ends with the scientific knowledge being used in society in, for instance, policy-making or product development. All discussions related to the research idea, presentations and seminars, online or offline, between the moment of getting the research idea and when the manuscript is published as a scientific publication, are part of either informal or formal scholarly communication. Although there are many aspects to scholarly communication, the formal scientific publication is crucial in every discipline. Kircz (1998) writes that "the scientific article is the object around which the whole fabric of writing, publishing, and reading is centered." While the scientific article is the repository of knowledge, citations place the article in time and connect it to earlier research. Figure 2.1
The cycle of scholarly communication.
By referencing earlier work the author gives credit to those before him and shows how his work builds upon that of other researchers. Bernal (1971) wrote that "the methods of the scientist would be of little avail if he had not at his disposal an immense stock of previous knowledge and experience." Bernal (1971) continues by describing the cumulative nature of science as "an ever-growing body of knowledge built of sequences of the reflections and ideas, but even more of the experience and actions, of a great stream of thinkers and workers." With that Bernal eloquently summarizes how a researcher is dependent on the work of those before him and how this intellectual debt or connection to earlier work is indicated through citations. Similarly, assuming that the publication is good enough, it will be used and recognized by other researchers who give credit to it by referencing it in new scientific publications, starting the process of scholarly communication all over again (Figure 2.1).
The references given and citations received are the links between earlier research and current research. They draw up the timeline and map developments of science. They are also part of the scientific reward system that stems from the...