Introduction

Research is an integral part of our world. It is responsible for, among other things, the medicines that we take, our economic policies, our approaches to marketing, the educational strategies used in our schools, therapeutic strategies for the mentally distressed and the techniques for harnessing energy for industry. This book is about evaluating research. It is about assessing studies that have shaped our lives in fundamental ways.

Research is to a great extent the domain of experts and specialists: scientists hired by governments and industries, professors in universities, PhDs working for foundations or in research centers. This book, however, is not for them, at least not in so far as they are experts. It does not discuss the fine points of evaluating the internal consistency of theories, nor the empirical coverage of different kinds of theoretical models. As important as all of these discussions are for the specialists engaged in research, they are not within the domain of this book.

This book is for the rest of us: those of us who read about research but are not experts in the field. It is for those of us who search for ways of understanding important ideas and try to identify poorly thought-out proposals in areas in which we have some interest or concerns, but in which we have not been trained to be specialists. Since even experts are only experts in limited areas, this means that this book is for all of us.

All of us need ways to assess the research findings we encounter. Whether we are reading about the newest medical discoveries, trying to decide where to invest our money, considering the proposals of politicians for the development of new energy resources - in all of those many areas in which research and research findings affect our lives and we have no way of making an expert assessment - all of us need a basic knowledge of what to look for, and what to look out for.

When research is reported in popular media outlets, its findings are often presented as established facts, and even sometimes as a clarion call for immediate action. Nevertheless, it sometimes turns out that these calls to action are based on questionable research. In the 1990s, for example, menopausal women were urged by such reputable sources as the American Heart Association (Felgran and Hettinger, 2002: 71) and the American College of Physicians (Kolata and Petersen, 2002) to have hormone replacement therapy (HRT) in order to decrease the chances of developing heart disease and osteoporosis. By 2002 HRT was viewed far more skeptically as a result of later findings: while it indeed lowered the risk of osteoporosis, at the same time it increased the risk of stroke, heart disease, breast cancer and dementia (Writing Group for the Women's Health Initiative Investigators, 2002).

The financial sphere is another example of an area where information, even when disseminated from reputable sources, requires close scrutiny. Enron was considered one of the fastest-growing companies and a safe investment. Anyone researching the company found reassuring financial statements based on apparently impeccable accounting research. Nevertheless, the collapse of Enron became worldwide news. It might be tempting to think that the Enron situation is an aberration, but the accounting conflicts of interest that contributed to it have been widespread (Fearnley et al., 2005: 69).

The intention of the book is to present the reader with some of the basic concepts that can be used for assessing the quality of research, regardless of its topic or area of specialization. It tries to point the reader to warning flags that ought to prompt further questions and perhaps some skepticism. In sum, this book is about assessing research as a non-expert in an intelligent, educated manner.

There are tried-and-true methods of deciding whether particular research is worth even considering. In the second chapter we discuss these criteria, showing why they are used and pointing out where they can be useful. However, we also show you where these criteria fall short, and why trusting these gold standards as the primary criteria for assessing research can lead to misplaced confidence.

In Chapters 3-5 we look at issues that can affect the validity of research. In particular, we look at questions related to the funding of research, the theoretical models upon which the research is based and the venues available for disseminating it. In each case we investigate the way in which economic and ideological systems can affect research and its outcomes.

In the third chapter we examine the issues of funding and sponsorship. We show how the sources that fund and sponsor research may affect its outcomes and conclusions. In so doing, we look at three different cases in quite different fields. First we examine the ways in which pharmaceutical funding and sponsorship of drug-related research have affected the way in which the findings of that research are viewed. We look at how pharmaceutical companies used funding to spin the research evidence to make hormone replacement therapies appear safer than the research might have otherwise suggested.

Next, we look at the Enron debacle to see how funding conflicts of interest can corrupt the investigations of auditors (in other words, their research) into a company's financial health. We discuss how funding played a decisive role in corrupting the auditing of Enron's financial status and thus allowed Enron to produce flawed financial reports - reports that were nevertheless thought to be sound because the auditors had investigated Enron's reporting practices and approved them.

We then move to the field of psychology and the appearance of a much-heralded book, The Bell Curve, which claimed to establish racial differences in IQ scores. Although it might seem that the funding of research would relate solely to economic self-interest, The Bell Curve illustrates how funding can be used to promote ideological agendas as well. We discuss how one of the authors was supported by foundations that have specific ideological agendas. We address the convergence between the agendas of the funding sources and the findings of the research. Our interest is not in whether or not the book's conclusions are 'true' or offensive, but rather whether or not this convergence should prompt questions about the research.

While financial conflicts of interest are the stuff of which magazine and newspaper articles are often made, there are other considerations which are far subtler but can nevertheless affect the nature of research. In the fourth chapter we look at research from the perspective of how dominant research models (paradigms) exert influence not only over the conclusions of research studies, but even over the questions that are asked, the hypotheses that are investigated and the subjects that are studied. Theoretical paradigms constitute the set of assumptions - that is ideas - that form the framework within which research takes place. The focus of the fourth chapter is thus on the relationship between ideological systems and research.

We look first at one of the most influential research applications around the world, IQ tests, and show that the research supporting the use of these tests and the claim that they identify inheritable traits depends on particular theoretical assumptions. Different assumptions, we suggest, lead to different conclusions about intelligence and the meaning of its inheritability. We argue that an awareness of alternative perspectives is important in critically evaluating this research.

The next case we look at involves the causes of ulcers. Research throughout much of the twentieth century was based on a theoretical model that emphasized the role of stress in the development of ulcers. Treatment regimens were routinely recommended on the basis of this assumption. But late in the twentieth century a radically different theoretical model of ulcers was proposed. This model met with considerable resistance because it differed substantially from the dominant theoretical paradigm. Nevertheless, the newer model eventually gained acceptance because it led to more effective treatments. In science, accepted paradigms are periodically overthrown and replaced. In critically assessing research, we suggest, it is prudent to keep in mind that theories that seem far-fetched today may become the established truth of tomorrow.

Science is hardly the only area in which dominant paradigms affect research. In our final case study in this chapter we look at the effect of canons on research topics in the humanities and fine arts. We show how for years the focus of the Anglo-American literary canon meant that little research investigated anyone but British male writers. In recent years academicians have discovered, however, that there was serious literature created by a far more diverse group of writers, whose work had been ignored because it was not part of the canon. A similar trend can be found in the area of fine arts, and we discuss this with particular reference to female artists. We discuss how the canon discourages research into subjects that are excluded from it.

In Chapter 5 we turn to the dissemination of research. In the first case study we show that since the second half of the twentieth century there has been an explosion of journals created by groups that did not view themselves as adequately represented by mainstream journals, frequently because of ideological reasons. The fields of...