1
The Sample Size Problem in Animal-Based Research

CHAPTER OUTLINE HEAD

1. 1.1 Organisation of the Book
2. References

Good Numbers Matter. This is especially true when animals are research subjects. Researchers are responsible for minimising both direct harms to research animals and the indirect harms that result from wasting animals in poor-quality studies (Reynolds 2021). The ethical use of animals in research is framed by the 'Three Rs' principles of Replacement, Reduction, and Refinement. Originating over 60?years ago (Russell and Burch 1959), the 3Rs strategy is framed by the premise that maximal information should be obtained for minimal harms. Harms are minimised by Replacement, methods or technologies that substitute for animals; Reduction, the methods using the fewest animals for the most robust and scientifically valid information; and Refinement, the methods that improve animal welfare through minimising pain, suffering, distress, and other harms (Graham and Prescott 2015).

The focus of this book is on Reduction and methods of 'right-sizing' experiments. A right-sized experiment is an optimal size for a study to achieve its objectives with the least amount of resources, including animals. However, simply minimising the total number of animals is not the same as right-sizing. A right-sized experiment has a sample size that is statistically, operationally, and ethically defensible (Box 1.1). This will mean compromising between the scientific objectives of the study, production of scientifically valid results, availability of resources, and the ethical requirement to minimise waste and suffering of research animals. Thus, sample size calculations are not a single calculation but a set of calculations, involving iteration through formal estimates, followed by reality checks for feasibility and ethical constraints (Reynolds 2019).

BOX 1.1 Right-Sizing Checklist

Statistically defensible: Are numbers verifiable? (Calculations)

• Outcome variable identified
• Difference to be detected
• Expected variation in response
• Number of groups
• Anticipated statistical test (if hypothesis tests used)
• All calculations shown

Operationally defensible: Are numbers feasible? (Resources)

• Qualified technical staff
• Time
• Space
• Resources
• Equipment
• Funding

Ethically defensible: Are numbers fit for purpose? (3Rs)

• Appropriate for study objectives?
• Reasonable number of groups?
• Are collateral losses accounted for and minimized?
• Are loss mitigation plans described?
• Are 3Rs strategies described?

Source: Adapted from Reynolds (2021).

Additional challenges to right-sizing experiments include those imposed by experimental design and biological variability (Box 1.2). In The Principles of Humane Experimental Technique (1959), Russell and Burch were very clear that Reduction is achieved by systematic strategies of experimentation rather than trial and error. In particular, they emphasised the role of the statistically based family of experimental designs and design principles proposed by Ronald Fisher, still relatively new at the time. Formal experimental designs customised to address the particular research question increase the experimental signal through the reduction of variation. Design principles that reduce bias, such as randomisation and allocation concealment (blinding) increase validity. These methods increase the amount of usable information that can be obtained from each animal (Parker and Browne 2014).

Although it has now been almost a century since Fisher-type designs were developed many researchers in biomedical sciences still seem unaware of their existence. Many preclinical studies reported in the literature consist of numerous two-group designs. However, this approach is both inefficient and inflexible and unsuited to exploratory studies with multiple explanatory variables (Reynolds 2022). Statistically based designs are rarely reported in the preclinical literature. In part, this is because the design of experiments is seldom taught in introductory statistics courses directed towards biomedical researchers.

BOX 1.2 Challenges for Right-Sizing Animal-Based Studies

• Ethics and welfare considerations. The three Rs Replacement, Reduction, and Refinement should be the primary driver of animal numbers.
• Experimental design. Animal-based research has no design culture. Clinical trial models are inappropriate for exploratory research. Multifactorial agriculture/industrial design may be more suitable in many cases, and they are unfamiliar to most researchers.
• Biological variability. Animals can display significant differences in responses to interventions, making it challenging to estimate an appropriate sample size.
• Cost and resource constraints. The financial cost of conducting animal-based research, including the cost of housing, caring for, and monitoring the animals, must be considered in estimates of sample size.

Power calculations are the gold standard for sample size justification. However, they are commonly misapplied, with little or no consideration of study design, type of outcome variable, or the purpose of the study. The most common power calculation is for two-group comparisons of independent samples. However, this is inappropriate when the study is intended to examine multiple independent factors and interactions. Power calculations for continuous variables are not appropriate for correlated observations or count data with high prevalence of zeros. Power calculations cannot be used at all when statistical inference is not the purpose of the study, for example, assessment of operational and ethical feasibility, descriptive or natural history studies, and species inventories.

Evidence of right-sizing is provided by a clear plan for sample size justification and transparent reporting of the number of all animals used in the study. This is why these items are part of best-practice reporting standards for animal research publications (Kilkenny et al. 2010, Percie du Sert et al. 2020 and are essential for the assessment of research reproducibility (Vollert et al. 2020). Unfortunately, there is little evidence that either sample size justification or sample size reporting has improved over the past decade. Most published animal research studies are underpowered and biased (Button et al. 2013, Henderson et al. 2013, Macleod et al. 2015) with poor validity (Würbel 2017, Sena and Currie 2019), severely limiting reproducibility and translation potential (Sena et al. 2010, Silverman et al. 2017). A recent cross-sectional survey of mouse cancer model papers published in high-impact oncology journals found that fewer than 2% reported formal power calculations, and less than one-third reported sample size per group. It was impossible to determine attrition losses, or how many experiments (and therefore animals) were discarded due to failure to achieve statistical significance (Nunamaker and Reynolds 2022). The most common sample size mistake is not performing any calculations at all (Fosgate 2009). Instead, researchers make vague and unsubstantiated statements such as 'Sample size was chosen because it is what everyone else uses' or 'experience has shown this is the number needed for statistical significance'. Researchers often game, or otherwise adjust, calculations to obtain a preferred sample size (Schultz and Grimes 2005, Fitzpatrick et al. 2018). In effect, these studies were performed without justification of the number of animals used.

Statistical thinking is both a mindset and a set of skills for understanding and making decisions based on data (Tong 2019). Reproducible data can only be obtained by sustained application of statistical thinking to all experimental processes: good laboratory procedure, standardised and comprehensive operating protocols, appropriate design of experiments, and methods of collecting and analysing data. Appropriate strategies of sample size justification are an essential component.

1.1 Organisation of the Book

This book is a guide to methods of approximating sample sizes. There will never be one number or approach, and sample size will be determined for the most part by study objectives and choice of the most appropriate statistically based study design. Although advanced statistical or mathematical skills are not required, readers are expected to have at least a basic course on statistical analysis methods and some familiarity with the basics of power and hypothesis testing. SAS code is provided in appendices at the end of each chapter and references to specific R packages in the text. It is strongly recommended that everyone involved in devising animal-based experiments take at least one course in the design of experiments, a topic not often covered by statistical analysis courses.

Figure 1.1 Overview of book organisation. For animal numbers to be justifiable (Are they feasible? appropriate? ethical? verifiable?), sample...