Crowdsourcing for Speech Processing
Applications to Data Collection, Transcription and Assessment
Will be published approx. on 5. April 2013
Book
Hardback
356 pages
978-1-118-35869-6 (ISBN)
Description
Provides an insightful and practical introduction to crowdsourcing as a means of rapidly processing speech data
Intended for those who want to get started in the domain and learn how to set up a task, what interfaces are available, how to assess the work, etc. as well as for those who already have used crowdsourcing and want to create better tasks and obtain better assessments of the work of the crowd. It will include screenshots to show examples of good and poor interfaces; examples of case studies in speech processing tasks, going through the task creation process, reviewing options in the interface, in the choice of medium (MTurk or other) and explaining choices, etc.
* Provides an insightful and practical introduction to crowdsourcing as a means of rapidly processing speech data.
* Addresses important aspects of this new technique that should be mastered before attempting a crowdsourcing application.
* Offers speech researchers the hope that they can spend much less time dealing with the data gathering/annotation bottleneck, leaving them to focus on the scientific issues.
* Readers will directly benefit from the book's successful examples of how crowd- sourcing was implemented for speech processing, discussions of interface and processing choices that worked and choices that didn't, and guidelines on how to play and record speech over the internet, how to design tasks, and how to assess workers.
Essential reading for researchers and practitioners in speech research groups involved in speech processing
More details
Product info
gebunden
Edition
1. Auflage
Language
English
Place of publication
New York
United States
Target group
Professional and scholarly
Product notice
sewn/stitched
Paper over boards
Dimensions
Height: 251 mm
Width: 172 mm
Thickness: 25 mm
Weight
676 gr
ISBN-13
978-1-118-35869-6 (9781118358696)
Schweitzer Classification
Other editions
Additional editions
Maxine Eskenazi | Gina-Anne Levow | Helen Meng
Crowdsourcing for Speech Processing
Applications to Data Collection, Transcription and Assessment
E-Book
02/2013
Wiley
€99.99
Available for download
Maxine Eskenazi | Gina-Anne Levow | Helen Meng
Crowdsourcing for Speech Processing
Applications to Data Collection, Transcription and Assessment
E-Book
02/2013
Wiley
€99.99
Available for download
Persons
Maxine Eskenazi, Carnegie Mellon University, USA
Dr. Eskenazi is Principal Systems Scientist at the Language Technologies Institute, Carnegie Mellon University, USA. She has authored over 100 scientific papers in the areas of computer assisted language learning and speech and spoken dialog systems. Her work has produced such systems as the Let's Go spoken dialog system and the REAP vocabulary tutor. She is also the founder and CTO of the Carnegie Speech Company.
Gina-Anne Levow, University of Washington, USA
Dr. Levow is currently an Assistant Professor in the Department of Linguistics, University of Washington, USA. Prior to joining the faculty at the University of Washington, she served on the faculty at the University of Chicago in the Department of Computer Science and as a Research Fellow at the University of Manchester, UK. She served on the Editorial Board of Computational Linguistics and as Associate Editor of ACM Transactions on Asian Language Processing.
Helen Meng, The Chinese University of Hong Kong, Hong Kong
Dr. Meng is Founder and Director of the Human-Computer Communications Laboratory at The Chinese University of Hong Kong, and is also the Founder and Co-Director of the Microsoft-CUHK Joint Laboratory for Human-Centric Computing and Interface Technologies, which was conferred the national status of the Ministry of Education of China (MoE) Key Laboratory in 2008. Prof. Meng also served as an Associate Dean (Research) of the Faculty of Engineering from 2006 to 2010. She serves as Editor-in-Chief of the IEEE Transactions on Audio, Speech and Language Processing.
Gabriel Parent, Amazon.com, USA
Gabriel Parent is a Software Development Engineer at Amazon.com working on solving natural language related problems. His main research focuses were human-computer interaction through spoken dialog systems and crowdsourcing.
David Suendermann, Baden-Wuerttemberg Cooperative State University, Germany
Dr. Sundermann is currently full Professor of Computer Science at the Baden-Wuerttemberg Cooperative State University, Stuttgart, Germany. He is also the Principal Speech Scientist of SpeechCycle, New York, USA which has been recognized by Deloitte as a "Technology Fast 500" company based on revenue growth. He has authored more than 70 publications and patents, including a book and six book chapters.
Dr. Eskenazi is Principal Systems Scientist at the Language Technologies Institute, Carnegie Mellon University, USA. She has authored over 100 scientific papers in the areas of computer assisted language learning and speech and spoken dialog systems. Her work has produced such systems as the Let's Go spoken dialog system and the REAP vocabulary tutor. She is also the founder and CTO of the Carnegie Speech Company.
Gina-Anne Levow, University of Washington, USA
Dr. Levow is currently an Assistant Professor in the Department of Linguistics, University of Washington, USA. Prior to joining the faculty at the University of Washington, she served on the faculty at the University of Chicago in the Department of Computer Science and as a Research Fellow at the University of Manchester, UK. She served on the Editorial Board of Computational Linguistics and as Associate Editor of ACM Transactions on Asian Language Processing.
Helen Meng, The Chinese University of Hong Kong, Hong Kong
Dr. Meng is Founder and Director of the Human-Computer Communications Laboratory at The Chinese University of Hong Kong, and is also the Founder and Co-Director of the Microsoft-CUHK Joint Laboratory for Human-Centric Computing and Interface Technologies, which was conferred the national status of the Ministry of Education of China (MoE) Key Laboratory in 2008. Prof. Meng also served as an Associate Dean (Research) of the Faculty of Engineering from 2006 to 2010. She serves as Editor-in-Chief of the IEEE Transactions on Audio, Speech and Language Processing.
Gabriel Parent, Amazon.com, USA
Gabriel Parent is a Software Development Engineer at Amazon.com working on solving natural language related problems. His main research focuses were human-computer interaction through spoken dialog systems and crowdsourcing.
David Suendermann, Baden-Wuerttemberg Cooperative State University, Germany
Dr. Sundermann is currently full Professor of Computer Science at the Baden-Wuerttemberg Cooperative State University, Stuttgart, Germany. He is also the Principal Speech Scientist of SpeechCycle, New York, USA which has been recognized by Deloitte as a "Technology Fast 500" company based on revenue growth. He has authored more than 70 publications and patents, including a book and six book chapters.
Content
Contents
List of Contributors xiii
Preface xv
1 An Overview 1
Maxine Eskénazi
1.1 Origins of Crowdsourcing 2
1.2 Operational Definition of Crowdsourcing 3
1.3 Functional Definition of Crowdsourcing 3
1.4 Some Issues 4
1.5 Some Terminology 6
1.6 Acknowledgments 6
References 6
2 The Basics 8
Maxine Eskénazi
2.1 An Overview of the Literature on Crowdsourcing for Speech Processing 8
2.2 Alternative Solutions 14
2.3 Some Ready-Made Platforms for Crowdsourcing 15
2.4 Making Task Creation Easier 17
2.5 Getting Down to Brass Tacks 17
2.6 Quality Control 29
2.7 Judging the Quality of the Literature 32
2.8 Some Quick Tips 33
2.9 Acknowledgments 33
References 33
Further reading 35
3 Collecting Speech from Crowds 37
Ian McGraw
3.1 A Short History of Speech Collection 38
3.2 Technology for Web-Based Audio Collection 43
3.3 Example: WAMI Recorder 49
3.4 Example: The WAMI Server 52
3.5 Example: Speech Collection on Amazon Mechanical Turk 59
3.6 Using the Platform Purely for Payment 65
3.7 Advanced Methods of Crowdsourced Audio Collection 67
3.8 Summary 69
3.9 Acknowledgments 69
References 70
4 Crowdsourcing for Speech Transcription 72
Gabriel Parent
4.1 Introduction 72
4.2 Transcribing Speech 73
4.3 Preparing the Data 80
4.4 Setting Up the Task 83
4.5 Submitting the Open Call 91
4.6 Quality Control 95
4.7 Conclusion 102
4.8 Acknowledgments 103
References 103
5 How to Control and Utilize Crowd-Collected Speech 106
Ian McGraw and Joseph Polifroni
5.1 Read Speech 107
5.2 Multimodal Dialog Interactions 111
5.3 Games for Speech Collection 120
5.4 Quizlet 121
5.5 Voice Race 123
5.6 Voice Scatter 129
5.7 Summary 135
5.8 Acknowledgments 135
References 136
6 Crowdsourcing in Speech Perception 137
Martin Cooke, Jon Barker, and Maria Luisa Garcia Lecumberri
6.1 Introduction 137
6.2 Previous Use of Crowdsourcing in Speech and Hearing 138
6.3 Challenges 140
6.4 Tasks 145
6.5 BigListen: A Case Study in the Use of Crowdsourcing to Identify Words in Noise 149
6.6 Issues for Further Exploration 167
6.7 Conclusions 169
References 169
7 Crowdsourced Assessment of Speech Synthesis 173
Sabine Buchholz, Javier Latorre, and Kayoko Yanagisawa
7.1 Introduction 173
7.2 Human Assessment of TTS 174
7.3 Crowdsourcing for TTS: What Worked and What Did Not 177
7.4 Related Work: Detecting and Preventing Spamming 193
7.5 Our Experiences: Detecting and Preventing Spamming 195
7.6 Conclusions and Discussion 212
References 214
8 Crowdsourcing for Spoken Dialog System Evaluation 217
Zhaojun Yang, Gina-Anne Levow, and Helen Meng
8.1 Introduction 217
8.2 Prior Work on Crowdsourcing: Dialog and Speech Assessment 220
8.3 Prior Work in SDS Evaluation 221
8.4 Experimental Corpus and Automatic Dialog Classification 225
8.5 Collecting User Judgments on Spoken Dialogs with Crowdsourcing 226
8.6 Collected Data and Analysis 230
8.7 Conclusions and Future Work 238
8.8 Acknowledgments 238
References 239
9 Interfaces for Crowdsourcing Platforms 241
Christoph Draxler
9.1 Introduction 241
9.2 Technology 242
9.3 Crowdsourcing Platforms 253
9.4 Interfaces to Crowdsourcing Platforms 261
9.5 Summary 278
References 278
10 Crowdsourcing for Industrial Spoken Dialog Systems 280
David Suendermann and Roberto Pieraccini
10.1 Introduction 280
10.2 Architecture 283
10.3 Transcription 287
10.4 Semantic Annotation 290
10.5 Subjective Evaluation of Spoken Dialog Systems 296
10.6 Conclusion 300
References 300
11 Economic and Ethical Background of Crowdsourcing for Speech 303
Gilles Adda, Joseph J. Mariani, Laurent Besacier, and Hadrien Gelas
11.1 Introduction 303
11.2 The Crowdsourcing Fauna 304
11.3 Economic and Ethical Issues 307
11.4 Under-Resourced Languages: A Case Study 316
11.5 Toward Ethically Produced Language Resources 322
11.6 Conclusion 330
Disclaimer 331
References 331
Index 335