Foundation Models for Natural Language Processing
Pre-trained Language Models Integrating Media
1st Edition
Published on 24. May 2023
Book
Paperback/Softback
XVIII, 436 pages
978-3-031-23192-6 (ISBN)
Description
This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts.
Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models.
After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI.
More details
Product info
Paperback
Series
Edition
1st ed. 2023
Language
English
Place of publication
Cham
Switzerland
Publishing group
Springer International Publishing
Target group
Professional and scholarly
Illustrations
112
13 s/w Abbildungen, 112 farbige Abbildungen, 112 farbige Tabellen
XVIII, 436 p. 125 illus., 112 illus. in color.
Dimensions
Height: 235 mm
Width: 155 mm
Thickness: 25 mm
Weight
686 gr
ISBN-13
978-3-031-23192-6 (9783031231926)
DOI
10.1007/978-3-031-23190-2
Schweitzer Classification
Other editions
Additional editions
Gerhard Paaß | Sven Giesselbach
Foundation Models for Natural Language Processing
Pre-trained Language Models Integrating Media
Book
05/2023
1st Edition
Springer
€53.49
Shipment within 15-20 days
Persons
Dr. Gerhard Paaß is a Lead Scientist at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS). With a background in Mathematics, he is a recognized expert in the field of Artificial Intelligence, particularly in the area of Natural Language Processing. Dr. Paaß has previously worked at UC Berkeley in California and the University of Technology in Brisbane. He has served as reviewer and conference chair at various international conferences, including NeurIPS, CIKM, ECML/PKDD, ICDM, and KDD, where he regularly is a member of the program committee. Dr. Paaß has received a "best paper" award on probabilistic logic and is the author of about 70 publications for international conferences and journals. Recently, he authored the book "Artificial Intelligence: What's Behind the Technology of the Future?" (in German). He is currently involved in the creation of a computer center for Foundation Models. Besides experimental research on Foundation Models, he holds lectures for Deep Learning and Natural Language Understanding at the University of Bonn and in industry.
Sven Giesselbach is the leader of the Natural Language Understanding (NLU) team at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), where he has specialized in Artificial Intelligence and Natural Language Processing. He and his team develop solutions in the areas of medical, legal and general document understanding, which in their core build upon Foundation Models. Sven Giesselbach is also part of the Competence Center for Machine Learning Rhine-Ruhr (ML2R), where he works as a research scientist and investigates Informed Machine Learning, a paradigm in which knowledge is injected into machine learning models, in conjunction with language modeling. He has published more than 10 papers on Natural Language Processing and Understanding, which focus on the creation of application-ready NLU systems and the integration of expert knowledge in various stages of the solution design. He led the development of the Natural Language Understanding Showroom, a platform for showcasing state-of-the-art Natural Language Understanding models. He regularly gives talks about NLU at summer schools, conferences and AI-Meetups.
Sven Giesselbach is the leader of the Natural Language Understanding (NLU) team at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), where he has specialized in Artificial Intelligence and Natural Language Processing. He and his team develop solutions in the areas of medical, legal and general document understanding, which in their core build upon Foundation Models. Sven Giesselbach is also part of the Competence Center for Machine Learning Rhine-Ruhr (ML2R), where he works as a research scientist and investigates Informed Machine Learning, a paradigm in which knowledge is injected into machine learning models, in conjunction with language modeling. He has published more than 10 papers on Natural Language Processing and Understanding, which focus on the creation of application-ready NLU systems and the integration of expert knowledge in various stages of the solution design. He led the development of the Natural Language Understanding Showroom, a platform for showcasing state-of-the-art Natural Language Understanding models. He regularly gives talks about NLU at summer schools, conferences and AI-Meetups.
Content
1 Introduction 1.1 Scope of the Book 1.2 Preprocessing of Text 1.3 Vector Space Models and Document Classification 1.4 Nonlinear Classifiers 1.5 Generating Static Word Embeddings 1.6 Recurrent Neural Networks 1.7 Convolutional Neural Networks 1.8 Summary 2 Pre-trained Language Models2.1 BERT: Self-Attention and Contextual Embeddings 2.2 GPT: Autoregressive Language Models
2.3 Transformer: Sequence-to-Sequence Translation
2.4 Training and Assessment of Pre-trained Language Models
3 Improving Pre-trained Language Models
3.2 Capturing Longer Dependencies
3.3 Multilingual Pre-trained Language Models
3.4 Additional Knowledge for Pre-trained Language Models
3.5 Changing Model Size
3.6 Fine-tuning for Specific Applications
4. Knowledge Acquired by Foundation Models 4.1 Benchmark Collections 4.2 Evaluating Knowledge by Probing Classifiers 4.3 Transferability and Reproducibility of Benchmarks 5 Foundation Models for Information Extraction 5.1 Text Classification5.2 Word Sense Disambiguation5.3 Named Entity Recognition 5.4 Relation Extraction 6 Foundation Models for Text Generation 6.1 Document Retrieval6.2 Question Answering 6.3 Neural Machine Translation 6.4 Text Summarization 6.5 Story Generation 6.6 Dialog Systems 7 Foundation Models for Speech, Images, Videos, and Control
7.1 Speech Recognition and Generation7.2 Image Processing and Generation 7.3 Video Interpretation and Generation 7.4 Controlling Dynamic Systems 8 Summary and Outlook 8.1 Foundation Models are a New Paradigm 8.2 Potential Harm from Foundation Models 8.3 Advanced Artificial Intelligence Systems Appendix
2.3 Transformer: Sequence-to-Sequence Translation
2.4 Training and Assessment of Pre-trained Language Models
3 Improving Pre-trained Language Models
3.2 Capturing Longer Dependencies
3.3 Multilingual Pre-trained Language Models
3.4 Additional Knowledge for Pre-trained Language Models
3.5 Changing Model Size
3.6 Fine-tuning for Specific Applications
4. Knowledge Acquired by Foundation Models 4.1 Benchmark Collections 4.2 Evaluating Knowledge by Probing Classifiers 4.3 Transferability and Reproducibility of Benchmarks 5 Foundation Models for Information Extraction 5.1 Text Classification5.2 Word Sense Disambiguation5.3 Named Entity Recognition 5.4 Relation Extraction 6 Foundation Models for Text Generation 6.1 Document Retrieval6.2 Question Answering 6.3 Neural Machine Translation 6.4 Text Summarization 6.5 Story Generation 6.6 Dialog Systems 7 Foundation Models for Speech, Images, Videos, and Control
7.1 Speech Recognition and Generation7.2 Image Processing and Generation 7.3 Video Interpretation and Generation 7.4 Controlling Dynamic Systems 8 Summary and Outlook 8.1 Foundation Models are a New Paradigm 8.2 Potential Harm from Foundation Models 8.3 Advanced Artificial Intelligence Systems Appendix