Publication, citation and bibliometric assessment of research

Abstract:

This chapter examines bibliometric assessments of research, including publication in general, journal lists and types of citation analysis, including individual citations and impact factors. We examine the central role of publication in research and the reasons for this, before examining methods of ranking publications. We discuss the benefits and limitations of citation analysis. The Excellence in Research Australia framework is investigated.

Key words

publication

bibliometrics

citation analysis

impact factor

journal list

Excellence in Research Australia

Publication - particularly publication in a refereed journal or book - is central to science and research. There are good reasons for this, as we examine in this chapter. The centrality of publication also means it is a good place to start when examining research assessment. Publication itself, and the processes of refereeing and review, are themselves forms of assessment. As such, perhaps the easiest way of assessing research would be simply to count publications. Funding could be linked directly to these counts. Indeed this method, albeit a highly flawed one, was used in Australia until quite recently. However, while publication might be one type of quality control, there are differences in the quality, relevance and so on of publications. There is considerable debate as to what type of publications are more important, with refereed articles dominant in the physical sciences, but with books challenging them in the humanities and some social sciences, depending on discipline and culture.

How this published research is evaluated and ranked is often seen to be of two major variants. On one hand, variants of peer and panel assessment of research or research portfolios, are seen in the United Kingdom and New Zealand and elsewhere. We examine these in subsequent chapters. On the other, 'quantitative' methods, sometimes termed 'metrics' or 'bibliometrics' can be used. This chapter will focus on the latter. Bibliometric methods can include count of publications in journals or other media, counts of articles as published in certain ranked or prestige journals, patents, and the use of citations, of which there are numerous measures. There are various methods of counting publications, including software such as the Web of Science citation indexes, Scopus and Google Scholar, and the self-reporting of publications. Citations are where one work acknowledges another, and they are recorded by various databases including the Web of Science, Scopus, Google Scholar and an increasing number of other online databases.

Advocates of bibliometric measures note their apparent ease of use, the cost advantages they may have over highly expensive peer and panel review processes, and even claim an 'objective element' where apparently unbiased citation and journal publication counts are spat out by various software packages and websites, rather than relying on the 'subjective' opinion of a group of panel members. In turn, critics claim bibliometrics are far from objective, favouring certain approaches (often claimed to be 'scientific' ones) and publication in journals rather than other fora, and favouring certain groups within the academe; while there are seen to be inherent problems with viewing publication and citations counts as evidence of quality.

This chapter will take a position somewhere between these two poles. First, it should be pointed out that the dichotomy between 'peer review' or panel review and bibliometric measures is a false one. Both are, to some extent, variants of peer review. To be published in a journal and some other outlets requires peer assessment, and having one's work cited is to some extent an acknowledgement of something by one's peers, although there might well be dispute about what is being acknowledged. Citations and publications are used in some measures of research performance in some panel assessments of research quality. There are drawbacks with both. Neither process is wholly objective or wholly unbiased, or without considerable limitations. There are problems with seeing bibliometrics, particularly citations, as simple measures of the quality of research, as we will discuss. It is also naive to see them as simple 'objective' measures without the subjectivity and power relationships that exist in other forms of research assessment. On the other hand, peer review panels are not simply groups of disinterested scholars politely judging research on its own merits; issues of bias, cronyism and power, and other problems arise too in these forms of research assessment, as we examine in other chapters. Both depend on the subjective and flawed judgement of humans, albeit possibly a greater pool of humans in the case of citations. We conclude by noting the difficult relationship between citations and quality, and urge caution in their use.

Publication and its centrality to research assessment

Publication is the standard measure of scientific, scholarly and research output. Given its central role in the fields of science and research, it deserves further investigation. Publication is a mark of ownership, to some extent a factor in noting and developing quality of research; it facilitates the exchange of ideas and research, and provides a marker in the history and development of a discipline (Kelly et al., 2009). It is the basis on which nearly all research assessment models are based. This section examines why publication has this central role.

Publication has a long history as the measure of science and achievement. A number of learned societies were founded in the seventeenth century, the Accademia del Cimento in Italy in 1657, the Royal Society in London in 1660, and the Academie Royale des Sciences in Paris in 1666; some in the eighteenth century, including the Linnean Society in 1788; but only a few existed before 1800, with an explosion of new societies from the nineteenth century onwards (Cohen et al., 1954). These worked to formalise and professionalise disciplines, with publication a key part of this process.

Publication has an important role in the development of science, social science, and the humanities. One view is that at the heart of science is the notion of a conversation. Publication allows for research and ideas to be 'out there' for discussion and critique, and perhaps adaption, modification and development. Scholarly societies promoted and formalised the meetings, presentations, publications and sharing of results and methods, and replication of studies, that developed with the professionalisation of science and scholarship. As refereeing became standard by the mid-twentieth century (see Chapter 3), this conversation became more formalised and subject to a form of quality control, albeit a highly flawed control. Not all contributions are deemed to be worthy of inclusion in this conversation, and indeed a large number of carefully written and refereed articles are probably not read by more than the reviewers and editors of journals, and are largely forgotten in the dustbin of history. This is not to say that only good ideas survive in some sort of evolutionary struggle, and perhaps some very good research is forgotten for reasons quite outside its quality. Some revolutionary ideas might be neglected for years, only to be revived. As such publication is a record of the development of a discipline - but it also provides a record of ideas that might be unjustly neglected or overlooked at the time of publication. One example is Mendel's foundational work on genetics, originally published in the mid-nineteenth century. Much was developed in a monastery (Mendel was a monk) and published in obscure journals, eventually to be rediscovered and celebrated after almost half a century had passed and Mendel had been dead for 15 years.

First publication is seen as a mark of ownership of ideas and discoveries. However, as recently as the debate between Newton and Leibniz on primacy of the creation of calculus, there was controversy as to whether first publication established the claim. Newton had almost certainly developed and discussed the ideas first, while Leibniz's version of calculus was the first to be published (Bardi, 2006). Primacy of publication, however, hardened over time as the most important claim to priority of ideas, research and discovery. For example, in the mid-nineteenth century Darwin had been working on his theory of evolution for years, but rushed The Origin of Species into print in 1859 after the great naturalist Alfred Wallace sent him a paper outlining his own, and similar, theory of evolution. Darwin presented both Wallace's papers and his own writings jointly to the Linnean Society, albeit without Wallace's knowledge. This was published under both their names as 'On the Tendency of Species to Form Varieties; and on the Perpetuation of Varieties and Species by Natural Means of Selection' in the Proceedings of the Linnean Society in 1858, but history talks of Darwinism not Wallacism. Wallace's contribution has been neglected until recent times - perhaps only underlying the importance of publishing first if one wants one's name enshrined in history. That Wallace's employment history was rather...