1
Introduction

Emmanuel Vincent Sharon Gannot and Tuomas Virtanen

Source separation and speech enhancement are core problems in the field of audio signal processing, with applications to speech, music, and environmental audio. Research in this field has accompanied technological trends, such as the move from landline to mobile or hands-free phones, the gradual replacement of stereo by 3D audio, and the emergence of connected devices equipped with one or more microphones that can execute audio processing tasks which were previously regarded as impossible. In this short introductory chapter, after a brief discussion of the application needs in Section 1.1, we define the problems of source separation and speech enhancement and introduce relevant terminology regarding the scenarios and the desired outcome in Section 1.2. We then present the general processing scheme followed by most source separation and speech enhancement approaches and categorize these approaches in Section 1.3. Finally, we provide an outline of the book in Section 1.4.

1.1 Why are Source Separation and Speech Enhancement Needed?

The problems of source separation and speech enhancement arise from several application needs in the context of speech, music, and environmental audio processing.

Real-world speech signals are often contaminated by interfering speakers, environmental noise, and/or reverberation. These phenomena deteriorate speech quality and, in adverse scenarios, speech intelligibility and automatic speech recognition (ASR) performance. Source separation and speech enhancement are therefore required in such scenarios. For instance, spoken communication over mobile phones or hands-free systems requires the separation or enhancement of the near-end speaker's voice with respect to interfering speakers and environmental noises before it is transmitted to the far-end listener. Conference call systems or hearing aids face the same problem, except that several speakers may be considered as targets. Source separation and speech enhancement are also crucial preprocessing steps for robust distant-microphone ASR, as available in today's personal assistants, car navigation systems, televisions, video game consoles, medical dictation devices, and meeting transcription systems. Finally, they are necessary components in providing humanoid robots, assistive listening devices, and surveillance systems with "super-hearing" capabilities, which may exceed the hearing capabilities of humans.

Besides speech, music and movie soundtracks are another important application area for source separation. Indeed, music recordings typically involve several instruments playing together live or mixed together in a studio, while movie soundtracks involve speech overlapped with music and sound effects. Source separation has been successfully used to upmix mono or stereo recordings to 3D sound formats and/or to remix them. It lies at the core of object-based audio coders, which encode a given recording as the sum of several sound objects that can then easily be rendered and manipulated. It is also useful for music information retrieval purposes, e.g. to transcribe the melody or the lyrics of a song from the separated singing voice.

This is an emerging research field with many real-life applications concerning the analysis of general sound scenes, involving the detection of sound events, their localization and tracking, and the inference of the acoustic environment properties.

1.2 What are the Goals of Source Separation and Speech Enhancement?

The goal of source separation and speech enhancement can be defined in layman's terms as that of recovering the signal of one or more sound sources from an observed signal involving other sound sources and/or reverberation. This definition turns out to be ambiguous. In order to address the ambiguity, the notion of source and the process leading to the observed signal must be characterized more precisely. In this section and in the rest of this book we adopt the general notations defined on p. xxv-xxvii.

1.2.1 Single-Channel vs. Multichannel

Let us assume that the observed signal has channels indexed by . By channel, we mean the output of one microphone in the case when the observed signal has been recorded by one or more microphones, or the input of one loudspeaker in the case when it is destined to be played back on one or more loudspeakers.1 A signal with channels is called single-channel and is represented by a scalar , while a signal with channels is called multichannel and is represented by an vector . The explanation below employs multichannel notation, but is also valid in the single-channel case.

1.2.2 Point vs. Diffuse Sources

Furthermore, let us assume that there are sound sources indexed by . The word "source" can refer to two different concepts. A point source such as a human speaker, a bird, or a loudspeaker is considered to emit sound from a single point in space. It can be represented as a single-channel signal. A diffuse source such as a car, a piano, or rain simultaneously emits sound from a whole region in space. The sounds emitted from different points of that region are different but not always independent of each other. Therefore, a diffuse source can be thought of as an infinite collection of point sources. The estimation of the individual point sources in this collection can be important for the study of vibrating bodies, but it is considered irrelevant for source separation or speech enhancement. A diffuse source is therefore typically represented by the corresponding signal recorded at the microphone(s) and it is processed as a whole.

1.2.3 Mixing Process

The mixing process leading to the observed signal can generally be expressed in two steps. First, each single-channel point source signal is transformed into an source spatial image signal (Vincent et al., 2012) by means of a possibly nonlinear spatialization operation. This operation can describe the acoustic propagation from the point source to the microphone(s), including reverberation, or some artificial mixing effects. Diffuse sources are directly represented by their spatial images instead. Second, the spatial images of all sources are summed to yield the observed signal called the mixture:

This summation is due to the superposition of the sources in the case of microphone recording or to explicit summation in the case of artificial mixing. This implies that the spatial image of each source represents the contribution of the source to the mixture signal. A schematic overview of the mixing process is depicted in Figure 1.1. More specific details are given in Chapter 3.

Note that target sources, interfering sources, and noise are treated in the same way in this formulation. All these signals can be either point or diffuse sources. The choice of target sources depends on the use case. Also, the distinction between interfering sources and noise may or may not be relevant depending on the use case. In the context of speech processing, these terms typically refer to undesired speech vs. nonspeech sources, respectively. In the context of music or environmental sound processing, this distinction is most often irrelevant and the former term is preferred to the latter.

Figure 1.1 General mixing process, illustrated in the case of sources, including three point sources and one diffuse source, and channels.

In the following, we assume that all signals are digital, meaning that the time variable is discrete. We also assume that quantization effects are negligible, so that we can operate on continuous amplitudes. Regarding the conversion of acoustic signals to analog audio signals and analog signals to digital, see, for example, Havelock et al. (2008, Part XII) and Pohlmann (1995, pp. 22-49).

1.2.4 Separation vs. Enhancement

The above mixing process implies one or more distortions of the target signals: interfering sources, noise, reverberation, and echo emitted by the loudspeakers (if any). In this context, source separation refers to the problem of extracting one or more target sources while suppressing interfering sources and noise. It explicitly excludes dereverberation and echo cancellation. Enhancement is more general, in that it refers to the problem of extracting one or more target sources while suppressing all types of distortion, including reverberation and echo. In practice, though, this term is mostly used in the case when the target sources are speech. In the audio processing literature, these two terms are often interchanged, especially when referring to the problem of suppressing both interfering speakers and noise from a speech signal. Note that, for either source separation or enhancement tasks, the extracted source(s) can be either the spatial image of the source or its direct path component, namely the delayed and attenuated version of the original source signal (Vincent et al., 2012; Gannot et al., 2001).

The problem of echo cancellation is out of the scope of this book. Please refer to Hänsler and Schmidt (2004) for a comprehensive overview of this topic. The problem of source localization and tracking cannot be viewed as a separation or enhancement task, but it is sometimes used as a preprocessing...