Handbook of Automated Scoring
Theory into Practice
1st Edition
Published on 9. March 2020
Book
Hardback
562 pages
978-1-138-57827-2 (ISBN)
Description
"Automated scoring engines [...] require a careful balancing of the contributions of technology, NLP, psychometrics, artificial intelligence, and the learning sciences. The present handbook is evidence that the theories, methodologies, and underlying technology that surround automated scoring have reached maturity, and that there is a growing acceptance of these technologies among experts and the public."
From the Foreword by Alina von Davier, ACTNext Senior Vice President
Handbook of Automated Scoring: Theory into Practice provides a scientifically grounded overview of the key research efforts required to move automated scoring systems into operational practice. It examines the field of automated scoring from the viewpoint of related scientific fields serving as its foundation, the latest developments of computational methodologies utilized in automated scoring, and several large-scale real-world applications of automated scoring for complex learning and assessment systems. The book is organized into three parts that cover (1) theoretical foundations, (2) operational methodologies, and (3) practical illustrations, each with a commentary. In addition, the handbook includes an introduction and synthesis chapter as well as a cross-chapter glossary.
From the Foreword by Alina von Davier, ACTNext Senior Vice President
Handbook of Automated Scoring: Theory into Practice provides a scientifically grounded overview of the key research efforts required to move automated scoring systems into operational practice. It examines the field of automated scoring from the viewpoint of related scientific fields serving as its foundation, the latest developments of computational methodologies utilized in automated scoring, and several large-scale real-world applications of automated scoring for complex learning and assessment systems. The book is organized into three parts that cover (1) theoretical foundations, (2) operational methodologies, and (3) practical illustrations, each with a commentary. In addition, the handbook includes an introduction and synthesis chapter as well as a cross-chapter glossary.
Reviews / Votes
'the Handbook of Automated Scoring is an excellent resource for understanding the theoretical, methodological and practical components of automated scoring. It provides a good foundation for understanding the considerations behind how assessments are designed and detailed methodological information about how to best create these kinds of systems. Part 3 that contains different illustrations of how to best design these systems is especially useful for students who are learning more about how these systems should work when implemented correctly.'- Magdalen Beiting-Parrish and Jay Verkuilen, International Statistical Review, 2021
More details
Series
Language
English
Place of publication
London
United Kingdom
Publishing group
Taylor & Francis Ltd
Target group
College/higher education
Professional and scholarly
Product notice
sewn/stitched
Cloth over boards
Illustrations
32 s/w Tabellen
32 Tables, black and white
Dimensions
Height: 236 mm
Width: 157 mm
Thickness: 36 mm
Weight
930 gr
ISBN-13
978-1-138-57827-2 (9781138578272)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Book
09/2021
1st Edition
Chapman & Hall/CRC
€76.80
Shipment within 10-20 days
E-Book
02/2020
1st Edition
Chapman & Hall/CRC
€68.49
Available for download
E-Book
02/2020
1st Edition
Chapman & Hall/CRC
€68.49
Available for download
Persons
Duanli Yan is Director of Data Analysis and Computational Research in the Psychometrics, Statistics, and Data Sciences area at the Educational Testing Service (ETS), and Adjunct Professor at Fordham University and Rutgers University. She is a co-author of Bayesian Networks in Educational Assessment and Computerized Adaptive and Multistage Testing with R, editor for Practical Issues and Solutions for Computerized Multistage Testing, and co-editor for Computerized Multistage Testing: Theory and Applications. Her awards include the 2016 AERA Division D Significant Contribution to Educational Measurement and Research Methodology Award.
Andre A. Rupp is Research Director in the Psychometrics, Statistics, and Data Sciences area at the Educational Testing Service (ETS). He is co-author and co-editor of two award-winning interdisciplinary books titled Diagnostic Measurement: Theory, Methods, and Applications and The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications. His synthesis- and framework-oriented research has appeared in a wide variety of prestigious peer-reviewed journals. He currently serves as the lead developer of the ITEMS professional development portal for NCME.
Peter W. Foltz is Vice President in Pearson's AI and Products Solutions Organization and Research Professor at the University of Colorado's Institute of Cognitive Science. His work covers machine learning and natural language processing for educational and clinical assessments, discourse processing, reading comprehension and writing skills, 21st-century skills learning, and large-scale data analytics. He has authored more than 150 journal articles, book chapters, and conference papers, as well as multiple patents.
Andre A. Rupp is Research Director in the Psychometrics, Statistics, and Data Sciences area at the Educational Testing Service (ETS). He is co-author and co-editor of two award-winning interdisciplinary books titled Diagnostic Measurement: Theory, Methods, and Applications and The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications. His synthesis- and framework-oriented research has appeared in a wide variety of prestigious peer-reviewed journals. He currently serves as the lead developer of the ITEMS professional development portal for NCME.
Peter W. Foltz is Vice President in Pearson's AI and Products Solutions Organization and Research Professor at the University of Colorado's Institute of Cognitive Science. His work covers machine learning and natural language processing for educational and clinical assessments, discourse processing, reading comprehension and writing skills, 21st-century skills learning, and large-scale data analytics. He has authored more than 150 journal articles, book chapters, and conference papers, as well as multiple patents.
Content
Contents
Foreword..................................................................................................................xi
1 The Past, Present, and Future of Automated Scoring..............................1
Peter W. Foltz, Duanli Yan, and Andre A. Rupp
Part I: Theoretical Foundations
2 Cognitive Foundations of Automated Scoring........................................ 13
Malcolm I. Bauer and Diego Zapata-Rivera
3 Assessment Design with Automated Scoring in Mind.........................29
Kristen DiCerbo, Emily Lai, and Matthew Ventura
4 Human Scoring with Automated Scoring in Mind................................49
Edward W. Wolfe
5 Natural Language Processing for Writing and Speaking..................... 69
Aoife Cahill and Keelan Evanini
6 Multimodal Analytics for Automated Assessment................................93
Sidney K. D'Mello
7 International Applications of Automated Essay Scoring.................... 113
Mark D. Shermis
8 Public Perception and Communication around Automated
Essay Scoring................................................................................................ 133
Scott W. Wood
9 An Evidentiary-Reasoning Perspective on Automated Scoring:
Commentary on Part I................................................................................ 151
Robert J. Mislevy
Part II: Operational Methodologies
10 Operational Human Scoring at Scale...................................................... 171
Kathryn L. Ricker-Pedley, Susan Hines, and Carolyn Connelly
11 System Architecture Design for Scoring and Delivery....................... 195
Sue Lottridge and Nick Hoefer
12 Design and Implementation for Automated
Scoring Systems........................................................................................... 217
Christina Schneider and Michelle Boyer
13 Quality Control for Automated Scoring in Large-Scale
Assessment.................................................................................................... 241
Dan Shaw, Brad Bolender, and Rick Meisner
14 A Seamless Integration of Human and Automated Scoring..............263
Kyle Habermehl, Aditya Nagarajan, and Scott Dooley
15 Deep Learning Networks for Automated Scoring Applications.......283
Saad M. Khan and Yuchi Huang
16 Validation of Automated Scoring Systems............................................ 297
Duanli Yan and Brent Bridgeman
17 Operational Considerations for Automated Scoring Systems:
Commentary on Part II............................................................................... 319
David M. Williamson
Part III: Practical Illustrations
18 Expanding Automated Writing Evaluation............................................ 329
Jill Burstein, Brian Riordan, and Daniel McCaffrey
19 Automated Writing Process Analysis.....................................................347
Paul Deane and Mo Zhang
20 Automated Scoring of Extended Spontaneous Speech.......................365
Klaus Zechner and Anastassia Loukina
21 Conversation-Based Learning and Assessment Environments.........383
Arthur C. Graesser, Xiangen Hu, Vasile Rus, and Zhiqiang Cai
22 Automated Scoring in Intelligent Tutoring Systems...........................403
Robert J. Mislevy, Duanli Yan, Janice Gobert, and Michael Sao Pedro
23 Scoring of Streaming Data in Game-Based Assessments...................423
Russell G. Almond
24 Automated Scoring in Medical Licensing..............................................445
Melissa J. Margolis and Brian E. Clauser
25 At the Birth of the Future: Commentary on Part III............................. 469
John T. Behrens
26 Theory into Practice: Reflections on the Handbook............................. 475
Andre A. Rupp, Peter W. Foltz, and Duanli Yan
Glossary................................................................................................................489
References............................................................................................................ 501
Index......................................................................................................................553
Foreword..................................................................................................................xi
1 The Past, Present, and Future of Automated Scoring..............................1
Peter W. Foltz, Duanli Yan, and Andre A. Rupp
Part I: Theoretical Foundations
2 Cognitive Foundations of Automated Scoring........................................ 13
Malcolm I. Bauer and Diego Zapata-Rivera
3 Assessment Design with Automated Scoring in Mind.........................29
Kristen DiCerbo, Emily Lai, and Matthew Ventura
4 Human Scoring with Automated Scoring in Mind................................49
Edward W. Wolfe
5 Natural Language Processing for Writing and Speaking..................... 69
Aoife Cahill and Keelan Evanini
6 Multimodal Analytics for Automated Assessment................................93
Sidney K. D'Mello
7 International Applications of Automated Essay Scoring.................... 113
Mark D. Shermis
8 Public Perception and Communication around Automated
Essay Scoring................................................................................................ 133
Scott W. Wood
9 An Evidentiary-Reasoning Perspective on Automated Scoring:
Commentary on Part I................................................................................ 151
Robert J. Mislevy
Part II: Operational Methodologies
10 Operational Human Scoring at Scale...................................................... 171
Kathryn L. Ricker-Pedley, Susan Hines, and Carolyn Connelly
11 System Architecture Design for Scoring and Delivery....................... 195
Sue Lottridge and Nick Hoefer
12 Design and Implementation for Automated
Scoring Systems........................................................................................... 217
Christina Schneider and Michelle Boyer
13 Quality Control for Automated Scoring in Large-Scale
Assessment.................................................................................................... 241
Dan Shaw, Brad Bolender, and Rick Meisner
14 A Seamless Integration of Human and Automated Scoring..............263
Kyle Habermehl, Aditya Nagarajan, and Scott Dooley
15 Deep Learning Networks for Automated Scoring Applications.......283
Saad M. Khan and Yuchi Huang
16 Validation of Automated Scoring Systems............................................ 297
Duanli Yan and Brent Bridgeman
17 Operational Considerations for Automated Scoring Systems:
Commentary on Part II............................................................................... 319
David M. Williamson
Part III: Practical Illustrations
18 Expanding Automated Writing Evaluation............................................ 329
Jill Burstein, Brian Riordan, and Daniel McCaffrey
19 Automated Writing Process Analysis.....................................................347
Paul Deane and Mo Zhang
20 Automated Scoring of Extended Spontaneous Speech.......................365
Klaus Zechner and Anastassia Loukina
21 Conversation-Based Learning and Assessment Environments.........383
Arthur C. Graesser, Xiangen Hu, Vasile Rus, and Zhiqiang Cai
22 Automated Scoring in Intelligent Tutoring Systems...........................403
Robert J. Mislevy, Duanli Yan, Janice Gobert, and Michael Sao Pedro
23 Scoring of Streaming Data in Game-Based Assessments...................423
Russell G. Almond
24 Automated Scoring in Medical Licensing..............................................445
Melissa J. Margolis and Brian E. Clauser
25 At the Birth of the Future: Commentary on Part III............................. 469
John T. Behrens
26 Theory into Practice: Reflections on the Handbook............................. 475
Andre A. Rupp, Peter W. Foltz, and Duanli Yan
Glossary................................................................................................................489
References............................................................................................................ 501
Index......................................................................................................................553