Handbook of Automated Scoring

Contents

Foreword..................................................................................................................xi

1 The Past, Present, and Future of Automated Scoring..............................1

Peter W. Foltz, Duanli Yan, and Andre A. Rupp

Part I: Theoretical Foundations

2 Cognitive Foundations of Automated Scoring........................................ 13

Malcolm I. Bauer and Diego Zapata-Rivera

3 Assessment Design with Automated Scoring in Mind.........................29

Kristen DiCerbo, Emily Lai, and Matthew Ventura

4 Human Scoring with Automated Scoring in Mind................................49

Edward W. Wolfe

5 Natural Language Processing for Writing and Speaking..................... 69

Aoife Cahill and Keelan Evanini

6 Multimodal Analytics for Automated Assessment................................93

Sidney K. D'Mello

7 International Applications of Automated Essay Scoring.................... 113

Mark D. Shermis

8 Public Perception and Communication around Automated

Essay Scoring................................................................................................ 133

Scott W. Wood

9 An Evidentiary-Reasoning Perspective on Automated Scoring:

Commentary on Part I................................................................................ 151

Robert J. Mislevy

Part II: Operational Methodologies

10 Operational Human Scoring at Scale...................................................... 171

Kathryn L. Ricker-Pedley, Susan Hines, and Carolyn Connelly

11 System Architecture Design for Scoring and Delivery....................... 195

Sue Lottridge and Nick Hoefer

12 Design and Implementation for Automated

Scoring Systems........................................................................................... 217

Christina Schneider and Michelle Boyer

13 Quality Control for Automated Scoring in Large-Scale

Assessment.................................................................................................... 241

Dan Shaw, Brad Bolender, and Rick Meisner

14 A Seamless Integration of Human and Automated Scoring..............263

Kyle Habermehl, Aditya Nagarajan, and Scott Dooley

15 Deep Learning Networks for Automated Scoring Applications.......283

Saad M. Khan and Yuchi Huang

16 Validation of Automated Scoring Systems............................................ 297

Duanli Yan and Brent Bridgeman

17 Operational Considerations for Automated Scoring Systems:

Commentary on Part II............................................................................... 319

David M. Williamson

Part III: Practical Illustrations

18 Expanding Automated Writing Evaluation............................................ 329

Jill Burstein, Brian Riordan, and Daniel McCaffrey

19 Automated Writing Process Analysis.....................................................347

Paul Deane and Mo Zhang

20 Automated Scoring of Extended Spontaneous Speech.......................365

Klaus Zechner and Anastassia Loukina

21 Conversation-Based Learning and Assessment Environments.........383

Arthur C. Graesser, Xiangen Hu, Vasile Rus, and Zhiqiang Cai

22 Automated Scoring in Intelligent Tutoring Systems...........................403

Robert J. Mislevy, Duanli Yan, Janice Gobert, and Michael Sao Pedro

23 Scoring of Streaming Data in Game-Based Assessments...................423

Russell G. Almond

24 Automated Scoring in Medical Licensing..............................................445

Melissa J. Margolis and Brian E. Clauser

25 At the Birth of the Future: Commentary on Part III............................. 469

John T. Behrens

26 Theory into Practice: Reflections on the Handbook............................. 475

Andre A. Rupp, Peter W. Foltz, and Duanli Yan

Glossary................................................................................................................489

References............................................................................................................ 501

Index......................................................................................................................553