1.0
Introduction to the Book

Multimodal discourse analysis is an area of research that is becoming more and more widely used in applied linguistics, sociolinguistics, education, psychology, anthropology, business and other applied social sciences. There are a number of multimodal approaches that have sprouted up over the past 20 years (Bateman 2008; Forceville 1994; Jewitt 2002; Kress and van Leeuwen 1996, 1998, 2001; van Leeuwen 1999; Mondada 2006; Norris 2002a, 2004a; O'Halloran 1999; O'Toole 1994; Scollon 1998, 2001a, b; Stöckl 2001). Yet, Multimodal (Inter)action Analysis (Norris 2002a, 2004a, 2011a, 2013a) is the only interdisciplinary approach that has been developed specifically for the analysis of multimodal action and interaction. In this approach, emphasis is placed upon the actions that people take as opposed to the language plus particular non-verbal movements that they produce (Goodwin 1981; Mondada 2014; Scollon 1979), the cognitive work that they do (Anderson 1990; Collins and Quillian 1969; Fodor 1975; Kintsch 1988; Newell and Simon 1976; Tulving 1983) or the psychological expressions that they display (Ekman 1979; Ekman and Friesen 1969). Actions, of course, are embodied and cognitive, psychological and performed with language plus non-verbal movements. In fact, all of the components, the verbal, non-verbal, environmental, cognitive, and psychological come together in our approach to analyzing multimodal (inter)action (Norris 2013a, b). Through systematic analysis of actions and interactions, this approach allows us to gain new insight into human action and interaction in a holistic and comprehensive way, and this book demonstrates how to engage in systematic analysis of multimodal (inter)actions.

What This Book is About

This book illustrates the phases and steps used when engaging in a Multimodal (Inter)action Analysis. The Step-by-Step process outlined here is a guide that shows how you can systematically work with multimodal data. This systematic guide consists of five phases, four of which consist of a number of steps each and one of which consist of a great number of analytical tools. The book emphasizes our working with video data. However, multimodal (inter)action analysts also use this process when working with different kinds of data that include but are not limited to video data. This is shown to some extent in the chapter sections that demonstrate how to use the Step-by-Step guide with examples from an experimental study and a video ethnography. However, because this book focuses upon video data and the holistic ways to analyze it, other data such as interviews or observational notes may be alluded to, but not worked with here in detail. However, it is important to note that for us, observational notes, text messages, emails, interviews, and diary entries also often are a part of the data collected and analyzed. But now, let us turn to what this book is about.

This book is written for undergraduate and graduate students as well as for emergent and established researchers wishing to engage in multimodal discourse analysis in a theoretically founded and interdisciplinary manner, integrating the verbal and non-verbal with object use and the embeddedness of people with the environment. Some sections of chapters are more geared towards undergraduate students, while other sections of chapters are more useful for Masters students, and again other sections are particularly important for PhD students and researchers wanting to engage in video ethnography. Some chapter sections are absolutely necessary to read for all readers, while other chapter sections are focused to this or that readership. Here, I would like to allude the reader to what is most necessary to engage in for whom to make reading choices and reading assignments easier and clearer for teachers and students alike. The gray boxes contain notes which give a quick overview of which chapters and sections are useful for whom and what to expect to find in them.

Systematically Working with Video Data: Phases I-V

Chapter 1: Systematically Working with Multimodal Data: Introduction

Note 1

Chapter 1 is a useful read, but not a must-read for undergraduate or graduate students starting out to learn how to conduct a multimodal discourse analysis. PhD students will definitely want to read Chapter 1.1.

Note 2

Chapter 1 introduces the reader to the book overall, presents a chapter-by-chapter outline, and offers the keen reader a vast amount of references in some of its paragraphs. Without going into too much detail, these references can be used as a guide to delve deeper into the background literature alluded to here.

Chapter 1, besides first giving a brief overview of what you can find in the chapters to come, quickly sums up other multimodal research areas and the literature background of Multimodal (Inter)action Analysis. Here, you will find an abundant amount of literature referenced and the deliberate reader may want to read some of the texts referred to here. The literature, although quite vast in these paragraphs, is not meant to be comprehensive. Other books and online references are a better place for comprehensive literature reviews (Norris 2015 a-e; Pirini 2017). Also, you will not find me lingering on what has been said or done in many of the referenced texts. Rather, I allude to them in order to lead you to other work if you are inclined to move deeper into the multimodality literature. Here, I wish to demonstrate some of the vastness that already exists in this area of relatively new research and guide the careful reader to see differences as well as similarities. When differences are pointed out, the reader needs to understand that differences are not negative, but rather simply developments from different theoretical thought. In fact, no method, including the one outlined in detail in this book, is perfect. No methods can analyze everything, and no method is necessarily better than another. Methods are tools and, just as you will use a different set of tools when you are working with jewelry than when you are building a house, you will use different tools to understand multimodal action and interaction than you will if you are interested in studying text-image relations. Just as jewelry tools are not better than tools that help you build a house, so are tools to study multimodal actions and interactions not better than tools that allow you to examine text-image relations. What is important is to use the right tools for what you intend to do. You will not do well with working on jewelry if you are using tools that are made to build a house. Similarly, you will lack ability to study human actions and interactions if you are using the tools made to study text-image relations. This book is about the tools that help you study the multimodality of human actions and interactions. Specifically, Chapter 2 guides you to first understand the theoretical thought behind why we think of multimodal action and interaction in the way outlined in this text.

Chapter 2: Philosophical and Theoretical Background

Note 3

Chapter 2 is an absolute must-read for all instructors, PhD students and researchers particularly up to (and including) the part on the site of engagement.

Note 4

The sections about the development of Scollon's philosophical thought are offered for the keen reader, who is inclined to wanting to know more about the origin of some of the philosophical and theoretical thoughts underpinning this book. This is not a must-read for undergraduate or Masters-level students, but useful for PhD-level students.

Note 5

Chapter 2 provides the theoretical underpinning for all to be learned in chapters to come. Without this theoretical understanding (up to and including the section on the site of engagement), the rest of the book will make relatively little sense, since all phases and steps outlined in the chapters to come build upon the philosophical and theoretical background offered in Chapter 2.

In Chapter 2, you find the philosophical underpinnings, which are the primacy of perception and the primacy of embodiment. Then, you find the theoretical underpinnings, the principle of social actions and the two sub-principles of communication and history, and an explanation of how the principle and sub-principles fit together. Besides the principles that underpin everything written in this book, Chapter 2 offers some basic theoretical concepts and links these to perception and embodiment. For example, here you find the definition and deep explanation of the unit of analysis, the mediated action. The mediated action, as you will see, consists of social actors (usually human beings) and mediational means/cultural tools (usually things, the environment, body parts, emotion, or knowledge). But of course, we can also have non-human actors. Just think of your dog or cat, and you can see how we could possibly see a pet as acting in similar ways as humans in our socialized world. But just as a dog may be thought of as a social actor, a human being can be...