CHAPTER 2
AI-based Forensic Evaluation in Court: The Desirability of Explanation and the Necessity of Validation

Rolf J.F. Ypma1, Daniel Ramos2, and Didier Meuwly1,3

1 Netherlands Forensic Institute, Netherlands
2 AUDIAS Lab. Universidad Autonoma de Madrid, Madrid, Spain
3 University of Twente, Enschede, Netherlands

2.1 Introduction

Artificial intelligence (AI) is based on complex algorithms, methods using them, and systems. In this chapter, we understand algorithms as the core technology (e.g., a deep neural network (DNN)), the methods as the application of that core technology to a particular problem (e.g., the use of a DNN to evaluate fingerprint evidence), and the system as the software tool(s) that implement that method and algorithm (e.g., the GUI software package, or API, that is commercialized). The core technology typically includes pattern recognition and machine learning algorithms, whose complexity and performance has increased dramatically over the last years. These algorithms are often explainable in their inputs and their outputs, but the rationale that governs their internal mechanisms can be very difficult to explain. A prime example is DNNs, a family of algorithms forming the so-called deep learning field (LeCun et al. 2015). A DNN is a dense and very complex grid of connections between units called neurons. These connections can even be recurrent, or complemented with filters that interrelate different regions in the input features. Originally, a neural network aimed at mimicking the topology of the human brain, hence its name. The input of a DNN is typically a list of numbers representing some textual or sensory raw data (e.g., a fingermark image or an audio file). The output of a DNN consists of continuous or discrete outputs, depending on the task (e.g., a probability for each of a set of classes in a classification task or a magnitude of interest to predict in a regression task). Thus, the inputs and outputs are well defined, and even explainable in many cases. However, it is very difficult to know or explain what the activation of any single neuron, or a group of them, means. Although some intuitions exist, interpreting those intermediate outcomes is extremely difficult in general. Indeed, the explainability of machine learning algorithms is a field in itself, named explainable AI (XAI), (Doshi-Velez and Kim 2017; Guidotti et al. 2018; LeCun et al. 2015; Molnar 2020) covering explainable algorithms for machine learning and AI. XAI is closely related to more philosophical areas such as ethics, fairness, and the risk of applicability of AI to real-world problems.

In the context of forensic evaluation using algorithms, we consider that to interpret means providing a forensic meaning to the results of the computation (e.g., a set of features or a score obtained or extracted from the evidence), and modelling them probabilistically in the form of a likelihood ratio. The likelihood ratio is a numerical expression of the probative value that is meaningful in the judicial context, where the defence and the prosecution alternative propositions are disputed (Evett et al. 2000; Jackson et al. 2013). However, the interpretation by humans in understandable terms of the inner components of the "black box" that forms many machine learning algorithms (such as DNNs) is still an issue, particularly for high-impact decisions as in forensic science. At this point it is worth highlighting that we distinguish between the interpretation of the inner components and workings of an AI algorithm, and the interpretation of its results (e.g., outputs such as the strength of evidence expressed as a likelihood ratio (LR) computed by an algorithm, or a class probability of a classifier). In the XAI literature these are often referred to as "global" and "local" explanations. The likelihood ratio framework is considered as a logical approach for the forensic interpretation of results of evidence evaluation, in this case in the context of a Bayesian decision scenario. But we can also think about the interpretation of the rationale of the methods themselves, i.e., how do they work internally.

Machine learning algorithms, when training has finished, are completely deterministic and thus reproducible and repeatable. This is good news when trying to characterize the performance and the behavior of a given algorithm in a given experimental set-up. Indeed, as it is described below, this repeatability and reproducibility makes the system testable, which is at the basis of a rigorous empirical validation process. However, in recent years the size and complexity of DNNs has steadily increased. As a consequence, creating explanations to provide insight into the rationale of the algorithm has become more challenging. We can, therefore, state that machine learning and pattern recognition algorithms in AI systems can be validated, but their rationale remains hard to explain.

As an example in the context of forensic science, it is common to use complex analytical chemistry methods for forensic examination. We can think about the use of laser ablation with inductively coupled plasma mass spectrometry (LA-ICP-MS) for glass comparative analysis. In court, it is extremely difficult to explain the process of extraction of information from glass with a LA-ICP-MS analytical device. As a result, explainability of the technical details of the method is not helpful in general, with lawyers generally lacking the competence to understand such an explanation. However, the results of the method are typically trusted and assumed to be reliable: courts generally accept forensic examination based on these chemical profiles as valuable evidence, as we think it should be. We believe that an important reason for this is the fact that the process of producing and comparing analytical results in forensic glass examination with LA-ICP-MS has been tested and validated according to international quality standards and accredited by a national accreditation body. This proves the method's reliability in court, more than any explanations that the forensic scientist could give to the judge or jury. In this case, explainability is welcome if it makes the whole process more understandable and transparent, but is not essential.

Explainability problems pile up if we consider a typical operational scenario, where the data have not been seen before, and may not have been well represented in the controlled dataset that was used to train the algorithms. It is very difficult to predict the behavior of a system trained on a controlled dataset in all situations. This is due to the complexity of the algorithm, but also to the diversity and variability of the possible scenarios. Even in controlled conditions, the complexity of the feature and parameter spaces of modern machine learning algorithms is huge. If the scenario in which the machine learning algorithm is going to operate is not very well characterized and targeted, the results might be unpredictable. The bad news is that, when this degradation of performance happens, it is very difficult to scrutinize the inner behavior of the algorithm in order to solve, or even explain, this lack of performance.

One recent problem that can help to understand this situation is the sensitivity of DNNs to adversarial noise. Lately, it was discovered that DNNs, although presenting extremely competitive performance in a wide variety of tasks, are also highly sensitive to so-called adversarial examples. Goodfellow et al. (2014) present a simple scenario in an image classification task, where a DNN is used to classify input images in classes (e.g., written digits, or type of objects in the image). The performance of the DNN was excellent in those tasks, where the conditions of the training and the testing datasets were similar. However, by adding so-called adversarial noise, i.e., a controlled degradation in the input image, the performance of the DNN dramatically dropped. The most intriguing fact about the experiment is that the adversarial noise could not be perceived by a human observer: the original and the noisy image looked exactly the same to the human eye. Also, this ability of adversarial noise to fool a DNN seems to manifest in different datasets and DNN architectures, revealing a vulnerability of DNNs for image classification in general. Of course, in forensic science it is difficult to manipulate data in order to add this kind of perturbation. However, the adversarial noise is a good example of potential unknown unknowns resulting in unexpected results from an AI-based system, illustrating our lack of understanding of such systems. The main message for forensic examiners is important: even if an algorithm such as a DNN has been tested in controlled conditions, it can fail in unexpected ways when dataset conditions are different.

Unfortunately, forensic conditions are always very variable, uncontrolled, and uncertain. This is a very challenging and unfavorable situation for any machine learning algorithm, but in particular it threatens the reliability of complex AI algorithms such as DNNs. Under such circumstances, the machine learning field has to continue to improve in order to generate solutions to a variety of scenarios where the operational data can be very variable and different to the training data. This includes strategies such as uncertainty incorporation to models, probabilistic calibration, domain adaptation, or transfer learning. All these approaches are aimed at making the system more robust to variation between datasets and data scarcity. Yet, safeguards at the operational level, such as a...