Deep Reinforcement Learning and Its Industrial Use Cases
AI for Real-World Applications
1st Edition
Published on 14. October 2024
Book
Hardback
416 pages
978-1-394-27255-6 (ISBN)
Description
This book serves as a bridge connecting the theoretical foundations of DRL with practical, actionable insights for implementing these technologies in a variety of industrial contexts, making it a valuable resource for professionals and enthusiasts at the forefront of technological innovation.
Deep Reinforcement Learning (DRL) represents one of the most dynamic and impactful areas of research and development in the field of artificial intelligence. Bridging the gap between decision-making theory and powerful deep learning models, DRL has evolved from academic curiosity to a cornerstone technology driving innovation across numerous industries. Its core premise-enabling machines to learn optimal actions within complex environments through trial and error-has broad implications, from automating intricate decision processes to optimizing operations that were previously beyond the reach of traditional AI techniques.
"Deep Reinforcement Learning and Its Industrial Use Cases: AI for Real-World Applications" is an essential guide for anyone eager to understand the nexus between cutting-edge artificial intelligence techniques and practical industrial applications. This book not only demystifies the complex theory behind deep reinforcement learning (DRL) but also provides a clear roadmap for implementing these advanced algorithms in a variety of industries to solve real-world problems. Through a careful blend of theoretical foundations, practical insights, and diverse case studies, the book offers a comprehensive look into how DRL is revolutionizing fields such as finance, healthcare, manufacturing, and more, by optimizing decisions in dynamic and uncertain environments.
This book distills years of research and practical experience into accessible and actionable knowledge. Whether you're an AI professional seeking to expand your toolkit, a business leader aiming to leverage AI for competitive advantage, or a student or academic researching the latest in AI applications, this book provides valuable insights and guidance. Beyond just exploring the successes of DRL, it critically examines challenges, pitfalls, and ethical considerations, preparing readers to not only implement DRL solutions but to do so responsibly and effectively.
Audience
The book will be read by researchers, postgraduate students, and industry engineers in machine learning and artificial intelligence, as well as those in business and industry seeking to understand how DRL can be applied to solve complex industry-specific challenges and improve operational efficiency.
Deep Reinforcement Learning (DRL) represents one of the most dynamic and impactful areas of research and development in the field of artificial intelligence. Bridging the gap between decision-making theory and powerful deep learning models, DRL has evolved from academic curiosity to a cornerstone technology driving innovation across numerous industries. Its core premise-enabling machines to learn optimal actions within complex environments through trial and error-has broad implications, from automating intricate decision processes to optimizing operations that were previously beyond the reach of traditional AI techniques.
"Deep Reinforcement Learning and Its Industrial Use Cases: AI for Real-World Applications" is an essential guide for anyone eager to understand the nexus between cutting-edge artificial intelligence techniques and practical industrial applications. This book not only demystifies the complex theory behind deep reinforcement learning (DRL) but also provides a clear roadmap for implementing these advanced algorithms in a variety of industries to solve real-world problems. Through a careful blend of theoretical foundations, practical insights, and diverse case studies, the book offers a comprehensive look into how DRL is revolutionizing fields such as finance, healthcare, manufacturing, and more, by optimizing decisions in dynamic and uncertain environments.
This book distills years of research and practical experience into accessible and actionable knowledge. Whether you're an AI professional seeking to expand your toolkit, a business leader aiming to leverage AI for competitive advantage, or a student or academic researching the latest in AI applications, this book provides valuable insights and guidance. Beyond just exploring the successes of DRL, it critically examines challenges, pitfalls, and ethical considerations, preparing readers to not only implement DRL solutions but to do so responsibly and effectively.
Audience
The book will be read by researchers, postgraduate students, and industry engineers in machine learning and artificial intelligence, as well as those in business and industry seeking to understand how DRL can be applied to solve complex industry-specific challenges and improve operational efficiency.
More details
Language
English
Place of publication
United States
Publishing group
John Wiley & Sons Inc
Target group
Professional and scholarly
Product notice
sewn/stitched
Cloth over boards
Dimensions
Height: 257 mm
Width: 183 mm
Thickness: 28 mm
Weight
916 gr
ISBN-13
978-1-394-27255-6 (9781394272556)
Copyright in bibliographic data and cover images is held by Nielsen Book Services Limited or by the publishers or by their respective licensors: all rights reserved.
Schweitzer Classification
Other editions
Additional editions
Shubham Mahajan | Pethuru Raj | Amit Kant Pandit
Deep Reinforcement Learning and Its Industrial Use Cases
AI for Real-World Applications
E-Book
10/2024
1st Edition
Wiley
€197.99
Available for download
Shubham Mahajan | Pethuru Raj | Amit Kant Pandit
Deep Reinforcement Learning and Its Industrial Use Cases
AI for Real-World Applications
E-Book
09/2024
1st Edition
Wiley
€197.99
Available for download
Persons
Shubham Mahajan, PhD, is an assistant professor in the School of Engineering at Ajeekya D Y Patil University, Pune, Maharashtra, India. He has eight Indian, one Australian, and one German patent to his credit in artificial intelligence and image processing. He has authored/co-authored more than 50 publications including peer-reviewed journals and conferences. His main research interests include image processing, video compression, image segmentation, fuzzy entropy, and nature-inspired computing methods with applications in optimization, data mining, machine learning, robotics, and optical communication.
Pethuru Raj, PhD, is chief architect and vice president at Reliance Jio Platforms Ltd in Bangalore, India. He has a PhD in computer science and automation from the Indian Institute of Science in Bangalore, India. His areas of interest focus on artificial intelligence, model optimization, and reliability engineering. He has published thirty research papers and edited forty-two books.
Amit Kant Pandit, PhD, is an associate professor in the School of Electronics & Communication Engineering Shri Mata Vaishno Devi University, India. He has authored/co-authored more than 60 publications including peer-reviewed journals and conferences. He has two Indian and one Australian patent to his credit in artificial intelligence and image processing. His main research interests are image processing, video compression, image segmentation, fuzzy entropy, and nature-inspired computing methods with applications in optimization.
Pethuru Raj, PhD, is chief architect and vice president at Reliance Jio Platforms Ltd in Bangalore, India. He has a PhD in computer science and automation from the Indian Institute of Science in Bangalore, India. His areas of interest focus on artificial intelligence, model optimization, and reliability engineering. He has published thirty research papers and edited forty-two books.
Amit Kant Pandit, PhD, is an associate professor in the School of Electronics & Communication Engineering Shri Mata Vaishno Devi University, India. He has authored/co-authored more than 60 publications including peer-reviewed journals and conferences. He has two Indian and one Australian patent to his credit in artificial intelligence and image processing. His main research interests are image processing, video compression, image segmentation, fuzzy entropy, and nature-inspired computing methods with applications in optimization.
Content
Preface xv
1 Deep Reinforcement Learning Applications in Real-World Scenarios: Challenges and Opportunities 1
Sunilkumar Ketineni and Sheela J.
1.1 Introduction 1
1.1.1 Problems with Real-World Implementation 2
1.2 Application to the Real World 3
1.2.1 Security and Robustness 3
1.2.2 Generalization 5
1.2.2.1 Overcoming Challenges in DRL 9
1.3 Possibilities for Making a Difference in the Real World 11
1.3.1 Transfer Learning and Domain Adaptation 11
1.4 Meta-Learning 12
1.5 Deep Reinforcement Learning (DRL) 13
1.5.1 Hybrid Approaches 14
1.6 Online vs. Offline Reinforcement Learning 15
1.7 Human-in-the-Loop Systems 15
1.8 Benchmarking and Standardization 16
1.9 Collaborative Multi-Agent Systems 18
1.10 Transfer Learning and Domain Adaptation 19
1.11 Hierarchical and Multimodal Learning 21
1.12 Imitation Learning and Human Feedback 22
1.13 Inverse Reinforcement Learning 23
1.14 Sim-to-Real Transfer 24
1.15 Conclusion 25
References 26
2 Deep Reinforcement Learning: A Key to Unlocking the Potential of Robotics and Autonomous Systems 29
Saksham and Chhavi Rana
2.1 Introduction 30
2.1.1 Significance of DRL Field 30
2.1.2 Transformative Advantages of DRL Field 32
2.2 Fields of Investigation 33
2.2.1 General Methods for Investigation 34
2.3 Background 36
2.3.1 Fundamentals of Deep Reinforcement Learning (DRL) 38
2.4 Deep Reinforcement Learning (DRL) in Robot Control 39
2.4.1 Navigation and Localization 40
2.4.2 Object Manipulation 42
2.5 Applications and Case Studies 43
2.6 Challenges and Future Directions 44
2.7 Evaluation and Metrics 46
2.8 Summary 47
References 48
3 Deep Reinforcement Learning Algorithms: A Comprehensive Overview 51
Shweta V. Bondre, Bhakti Thakre, Uma Yadav and Vipin D. Bondre
3.1 Introduction 52
3.1.1 How Reinforcement Learning Works? 53
3.2 Reinforcement Learning Algorithms 53
3.2.1 Value-Based Algorithms 53
3.2.1.1 Q-Learning 53
3.2.1.2 Deep Q-Networks (DQN) 57
3.2.1.3 Double DQN 58
3.2.1.4 Dueling DQN 58
3.3 Policy-Based 59
3.3.1 Policy Gradient Methods 59
3.3.2 REINFORCE (Monte Carlo Policy Gradient) 60
3.3.3 Actor-Critic Methods 61
3.3.4 Natural Policy Gradient Methods 62
3.4 Model-Based Reinforcement Learning 63
3.4.1 Probabilistic Ensembles with Trajectory Sampling (PETS) 63
3.4.2 Probabilistic Inference for Learning Control (PILCO) 64
3.4.3 Model Predictive Control (MPC) 65
3.4.4 Model-Agnostic Meta-Learning (MAML) 66
3.4.5 Soft Actor-Critic with Model Ensemble 67
3.4.6 Deep Deterministic Policy Gradients with Model (DDPG with Model) 68
3.5 Characteristics of Reinforcement Learning 69
3.6 DRL Algorithms and Their Advantages and Drawbacks 71
3.7 Conclusion 72
References 72
4 Deep Reinforcement Learning in Healthcare and Biomedical Applications 75
Balakrishnan D., Aarthy C., Nandhagopal Subramani, Venkatesan R. and Logesh T. R.
4.1 Introduction 76
4.2 Related Works 76
4.3 Deep Reinforcement Learning Framework 80
4.4 Deep Reinforcement Learning Applications in Healthcare and Biomedicine 81
4.5 Deep Reinforcement Learning Employs Efficient Algorithms 82
4.5.1 Deep Q-Networks 82
4.5.2 Policy Differentiation Techniques 82
4.5.3 Hindsight Experience Replay (HER) 82
4.5.4 Curiosity-Driven Exploration 82
4.5.5 Long Short-Term Memory Networks and Recurring Neural Network Designs 82
4.5.6 Multi-Agent DRL 83
4.6 Semi-Autonomous Control Based on Deep Reinforcement Learning for Robotic Surgery 83
4.6.1 Double Deep Q-Network (DDQN) 83
4.6.2 Materials and Methods 84
4.6.3 Results 86
4.6.4 Discussion 87
4.7 Conclusion 87
References 88
5 Application of Deep Reinforcement Learning in Adversarial Malware Detection 91
Manju and Chhavi Rana
5.1 Introduction 91
5.1.1 Background 95
5.1.2 Significance of Malware Detection 96
5.1.3 Challenges with Adversarial Attacks 96
5.2 Foundations of Deep Reinforcement Learning 97
5.2.1 Overview of Deep Reinforcement Learning 98
5.2.2 Core Concepts and Components 99
5.2.3 Relevance to Malware Detection 100
5.3 Malware Detection Landscape 101
5.3.1 Evolution of Malware Detection Techniques 102
5.3.2 Adversarial Attacks in Cybersecurity 103
5.3.3 Need for Advanced Detection Strategies 104
5.4 Deep Reinforcement Learning Techniques 104
5.4.1 Application of Deep Learning in Malware Detection 105
5.4.2 Reinforcement Learning Algorithms 106
5.5 Feature Selection Strategies 107
5.5.1 Importance of Feature Selection in Malware Detection 108
5.5.2 Techniques for Feature Selection 108
5.5.3 Optimization for Deep Reinforcement Learning Models 109
5.6 Datasets and Evaluation 110
5.7 Generating Adversarial Samples 111
Conclusion and Future Directions 112
Future Directions 112
References 112
6 Artificial Intelligence in Blockchain and Smart Contracts for Disruptive Innovation 115
Eashwar Sivakumar, Kiran Jot Singh and Paras Chawla
6.1 Introduction 115
6.1.1 Smart Contract 116
6.2 Literature Review 117
6.2.1 Blockchain and Smart Contracts in Digital Identity 117
6.2.2 Blockchain and Smart Contracts in Financial Security 118
6.2.3 Blockchain and Smart Contracts in Supply Chain Management 119
6.2.4 Blockchain and Smart Contracts in Insurance 120
6.2.5 Blockchain and Smart Contracts in Healthcare 121
6.2.6 Blockchain and Smart Contracts in Agriculture 121
6.2.7 Blockchain and Smart Contracts in Real Estate 122
6.2.8 Blockchain and Smart Contracts in Education and Research 123
6.2.9 Blockchain and Smart Contracts in Other Sectors 124
6.3 Critical Analysis of the Review 125
6.4 Blockchain and Artificial Intelligence 128
6.5 Discussion on the Reasoning for Implementation of Blockchain 129
6.6 Conclusion 130
References 130
7 Clinical Intelligence: Deep Reinforcement Learning for Healthcare and Biomedical Advancements 137
Keerthika K., Kannan M. and T. Saravanan
7.1 Introduction 138
7.2 Deep Reinforcement Learning Methods 138
7.2.1 Model-Free Methods 138
7.2.2 Policy Gradient Methods 139
7.2.3 Model-Based Methods 139
7.3 Applications of DRL in Healthcare 140
7.3.1 Tailored Treatment Recommendations 140
7.3.2 Optimization of Clinical Trials 141
7.3.3 Disease Diagnosis Support 142
7.3.4 Accelerated Drug Discovery and Design 142
7.3.5 Enhanced Robotic Surgery and Assistance 142
7.3.6 Health Management System 143
7.4 Challenges 143
7.5 Healthcare Data Types 144
7.5.1 Electronic Healthcare Records (EHRs) 144
7.5.2 Laboratory Data 145
7.5.3 Sensor Data 145
7.5.4 Biomedical Imaging Information 145
7.6 Guidelines for the Application of DRL 147
7.7 A Case Study: DRL in Healthcare and Biomedical Applications 147
7.7.1 Optimizing Radiation Therapy Dose Distribution in Cancer Treatment 147
7.7.2 Dose Strategy Model in Sepsis Patient Treatment 148
References 149
8 Cultivating Expertise in Deep and Reinforcement Learning Principles 151
Chilakalapudi Malathi and J. Sheela
8.1 Introduction 151
8.1.1 Reinforcement Learning's Constituent Parts 152
8.1.2 Process of Markov Decisions (MDP) 152
8.1.3 Learning Reinforcement Methods 153
8.2 Intensive Learning Foundations 164
8.2.1 A Definition of Deep Learning 164
8.2.2 Deep Learning Elements 164
8.2.2.1 Different Kinds of Deep Learning Networks 165
8.3 Integrating Deep Learning and Reinforcement Learning 172
8.3.1 Deep Reinforcement Learning 172
8.3.2 Deep Reinforcement Learning Complexity Problems 174
Conclusion 175
References 175
9 Deep Reinforcement Learning in Healthcare and Biomedical Research 179
Shruti Agrawal and Pralay Mitra
9.1 Introduction 180
9.1.1 Reinforcement Learning 180
9.1.2 Deep Reinforcement Learning 181
9.2 Learning Methods in Bioinformatics with Applications in Healthcare and Biomedical Research 182
9.2.1 Protein Folding 182
9.2.2 Protein Docking 183
9.2.3 Protein-Ligand Binding 185
9.2.4 Binding Peptide Generation 187
9.2.5 Protein Design and Engineering 188
9.2.6 Drug Discovery and Development 190
9.3 Applications in Biological Data 192
9.3.1 Omics Data 192
9.3.2 Medical Imaging 192
9.3.3 Brain/Body-Machine Interfaces 193
9.4 Adaptive Treatment Approach in Healthcare 193
9.5 Diagnostic Tools in Healthcare and Biomedical Research 195
9.6 Scope of Deep Reinforcement Learning in Healthcare and Biomedical Applications 196
9.6.1 State and Action Space 196
9.6.2 Reward 197
9.6.3 Policy 198
9.6.4 Model Training 199
9.6.5 Exploration 199
9.6.6 Credit Assignment 200
9.7 Conclusions 200
References 201
10 Deep Reinforcement Learning in Robotics and Autonomous Systems 207
Uma Yadav, Shweta V. Bondre and Bhakti Thakre
10.1 Introduction 208
10.2 The Promise of Deep Reinforcement Learning (DRL) in Real-World Robotics 210
10.3 Preliminaries 211
10.4 Enhancing RL for Real-World Robotics 222
10.5 Reinforcement Learning for Various Robotic Applications 224
10.6 Problems Faced in RL for Robotics 231
10.7 RL in Robotics: Trends and Challenges 232
10.8 Conclusion 235
References 236
11 Diabetic Retinopathy Detection and Classification Using Deep Reinforcement Learning 239
H.R. Manjunatha and P. Sathish
11.1 Introduction 239
11.2 Literature Survey 243
11.3 Diabetic Retinopathy Detection and Classification 248
11.4 Result Analysis 256
11.5 Conclusion 260
References 260
12 Early Brain Stroke Detection Based on Optimized Cuckoo Search Using LSTM-Gated Multi-Perceptron Neural Network 265
Anita Venaik, Asha A., Dhiyanesh B., Kiruthiga G., Shakkeera L. and Vinodkumar Jacob
12.1 Introduction 266
12.2 Literature Survey 268
12.2.1 Problem Statement 269
12.3 Proposed Methodology 270
12.3.1 Dataset Collection 270
12.3.2 Preprocessing 271
12.3.3 Genetic Feature Sequence Algorithm (GFSA) 275
12.3.4 Disease-Prone Factor (DPF) 281
12.3.5 Decision Tree-Optimized Cuckoo Search (DTOCS) 284
12.3.6 Long Short-Term Memory Gate Multilayer Perceptron Neural Network (LSTM-MLPNN) 289
12.4 Result and Discussion 293
12.4.1 Performance Matrix 293
12.5 Conclusion 296
References 297
13 Hybrid Approaches: Combining Deep Reinforcement Learning with Other Techniques 301
M. T. Vasumathi, Manju Sadasivan and Aurangjeb Khan
13.1 Introduction 302
13.1.1 Digital Twin-Introduction 302
13.1.2 Model of a Digital Twin 302
13.1.2.1 Steps Involved in Building a Digital Twin Prototype 303
13.1.3 Application Areas of Digital Twins 303
13.1.3.1 Digital Twin in Medical Field 304
13.1.3.2 Digital Twin in Smart City 304
13.1.3.3 Digital Twin in Sports 304
13.1.3.4 Digital Twin in Smart Manufacturing 305
13.2 Digital Twin Technologies 305
13.2.1 Data Acquisition and Sensors 306
13.2.2 Data Analytics and Machine Learning 306
13.2.3 Cloud Computing 307
13.2.4 Other Technologies 307
13.3 Integration of RL and Digital Twin 307
13.3.1 Motivation for Combining Digital Twin and RL 309
13.3.2 How RL Enhances Decision-Making Within Digital Twins 310
13.4 Challenges of Using RL in Digital Twins 311
13.5 Digital Twin Modeling with RL 312
13.6 Technology Underlying RL-Based Digital Twins 314
13.6.1 Integration of RL with Digital Twins in Four Stages 314
13.6.2 Tools and Libraries for Developing RL-Based Digital Twins 314
13.6.2.1 Simulation and Digital Twin Platforms 314
13.6.2.2 Reinforcement Learning Libraries 315
13.6.3 Integration with Existing Systems and IoT Devices for RL Deployment 315
13.6.3.1 Data Collection and Sensor Integration 315
13.6.3.2 Communication and Data Ingestion 316
13.6.3.3 Digital Twin Integration 316
13.6.3.4 RL Integration 316
13.6.3.5 Control and Actuation 316
13.6.3.6 Implementation of Feedback and Learning Process 316
13.6.3.7 Dashboard for Alert and Visualization 316
13.6.3.8 Ensuring the Security and Authentication 317
13.7 Industry-Specific Applications: A Case Study of DT in a Car Manufacturing Unit 317
13.7.1 IoT Components Required for Creating Digital Twin for the Manufacturing Unit 318
13.7.2 Architecture of the Proposed Digital Twin for Car Manufacturing Unit 318
13.7.3 Challenges and Opportunities in the Implementation of DTs for Car Manufacturing 320
13.8 Conclusion 321
References 322
14 Predictive Modeling of Rheumatoid Arthritis Symptoms: A High-Performance Approach Using HSFO-SVM and UNET-CNN 325
Anusuya V., Baseera A., Dhiyanesh B., Parveen Begam Abdul Kareem and Shanmugaraja P.
14.1 Introduction 326
14.1.1 Novelty of the Research 327
14.2 Related Work 328
14.2.1 Challenges and Problem Identification Factor 331
14.3 HSFO-SVM Based on LSTM-Gated Convolution Neural Network (lstmg-cnn) 332
14.3.1 C-Score and Cross-Fold Validation 332
14.3.2 Honey Scout Forager Optimization 335
14.3.3 Feature Selection Using SVM 336
14.3.4 UNET-CNN Classification 338
14.4 Result and Discussion 341
14.5 Conclusion 345
References 346
15 Using Reinforcement Learning in Unity Environments for Training AI Agent 349
Geetika Munjal and Monika Lamba
15.1 Introduction 349
15.2 Literature Review 351
15.3 Machine Learning 352
15.3.1 Categorization of Machine Learning 352
15.3.1.1 Supervised Learning 352
15.3.1.2 Unsupervised Learning 353
15.3.1.3 Reinforcement Learning 353
15.3.2 Classifying on the Basis of Envisioned Output 353
15.3.2.1 Classification 354
15.3.2.2 Regression 354
15.3.2.3 Clustering 354
15.3.3 Artificial Intelligence 354
15.4 Unity 354
15.4.1 Unity Hub 355
15.4.2 Unity Editor 355
15.4.3 Inspector 355
15.4.4 Game View 355
15.4.5 Scene View 355
15.4.6 Hierarchy 355
15.4.7 Project Window 356
15.5 Reinforcement Learning and Supervised Learning 356
15.5.1 Positive Reinforcement 357
15.5.2 Negative Reinforcement 357
15.5.3 Model-Free and Model-Based RL 357
15.6 Proposed Model 359
15.6.1 Setting Up a Virtual Environment 359
15.6.2 Setting Up of the Environment 360
15.6.2.1 Creating and Allocating Scripts for the Environment 361
15.6.2.2 Creating a Goal for the Agent 361
15.6.2.3 Reward-Driven Behavior 361
15.7 Markov Decision Process 362
15.8 Model-Based RL 362
15.9 Experimental Results 363
15.9.1 Machine Learning Models Used for the Environments 363
15.9.2 PushBlock 363
15.9.3 Hallway 365
15.9.4 Screenshots of the PushBlock Environment 368
15.9.5 Screenshots of the Hallway Environment 369
15.10 Conclusion 372
References 372
16 Emerging Technologies in Healthcare Systems 375
Ravi Kumar Sachdeva, Priyanka Bathla, Samriti Vij, Dishika, Madhur Jain, Lokesh Kumar, G. S. Pradeep Ghantasala and Rakesh Ahuja
16.1 Introduction 375
16.2 Personalized Medicine 376
16.3 AI and ML in Healthcare Sector 377
16.3.1 AI in Medical Diagnosis 378
16.3.2 Drug Discovery 378
16.3.3 Personalized Treatment Plans 379
16.3.4 Pattern Matching or Trend Detection 380
16.4 Immunotherapy 380
16.4.1 Monoclonal Antibodies 381
16.4.2 Checkpoint Inhibitors 381
16.4.3 CAR-T Cell Therapy 381
16.5 Regenerative Medicine 381
16.6 Digital Health (Use of Technology in Healthcare) 383
16.6.1 Wearable Devices 383
16.6.2 Telemedicine 384
16.6.3 Electronic Health Records 384
16.7 Health Inequity 385
16.7.1 Health Disparity 385
16.7.2 Health Equity 385
16.8 Future Directions in Healthcare Research 385
16.9 Challenges and Recommendations for Advanced Level of Modern Healthcare Technologies 386
16.9.1 Challenges 387
16.9.2 Recommendations 388
16.10 Healthcare Sector in Developing and Underdeveloped Countries 388
16.10.1 Healthcare Sector in Developing Countries 388
16.10.2 Healthcare Sector in Underdeveloped Countries 389
16.11 Comparison of Recent Progress and Future Mentoring in Healthcare Using Technology 389
16.12 Conclusion 391
References 392
Index 395
1 Deep Reinforcement Learning Applications in Real-World Scenarios: Challenges and Opportunities 1
Sunilkumar Ketineni and Sheela J.
1.1 Introduction 1
1.1.1 Problems with Real-World Implementation 2
1.2 Application to the Real World 3
1.2.1 Security and Robustness 3
1.2.2 Generalization 5
1.2.2.1 Overcoming Challenges in DRL 9
1.3 Possibilities for Making a Difference in the Real World 11
1.3.1 Transfer Learning and Domain Adaptation 11
1.4 Meta-Learning 12
1.5 Deep Reinforcement Learning (DRL) 13
1.5.1 Hybrid Approaches 14
1.6 Online vs. Offline Reinforcement Learning 15
1.7 Human-in-the-Loop Systems 15
1.8 Benchmarking and Standardization 16
1.9 Collaborative Multi-Agent Systems 18
1.10 Transfer Learning and Domain Adaptation 19
1.11 Hierarchical and Multimodal Learning 21
1.12 Imitation Learning and Human Feedback 22
1.13 Inverse Reinforcement Learning 23
1.14 Sim-to-Real Transfer 24
1.15 Conclusion 25
References 26
2 Deep Reinforcement Learning: A Key to Unlocking the Potential of Robotics and Autonomous Systems 29
Saksham and Chhavi Rana
2.1 Introduction 30
2.1.1 Significance of DRL Field 30
2.1.2 Transformative Advantages of DRL Field 32
2.2 Fields of Investigation 33
2.2.1 General Methods for Investigation 34
2.3 Background 36
2.3.1 Fundamentals of Deep Reinforcement Learning (DRL) 38
2.4 Deep Reinforcement Learning (DRL) in Robot Control 39
2.4.1 Navigation and Localization 40
2.4.2 Object Manipulation 42
2.5 Applications and Case Studies 43
2.6 Challenges and Future Directions 44
2.7 Evaluation and Metrics 46
2.8 Summary 47
References 48
3 Deep Reinforcement Learning Algorithms: A Comprehensive Overview 51
Shweta V. Bondre, Bhakti Thakre, Uma Yadav and Vipin D. Bondre
3.1 Introduction 52
3.1.1 How Reinforcement Learning Works? 53
3.2 Reinforcement Learning Algorithms 53
3.2.1 Value-Based Algorithms 53
3.2.1.1 Q-Learning 53
3.2.1.2 Deep Q-Networks (DQN) 57
3.2.1.3 Double DQN 58
3.2.1.4 Dueling DQN 58
3.3 Policy-Based 59
3.3.1 Policy Gradient Methods 59
3.3.2 REINFORCE (Monte Carlo Policy Gradient) 60
3.3.3 Actor-Critic Methods 61
3.3.4 Natural Policy Gradient Methods 62
3.4 Model-Based Reinforcement Learning 63
3.4.1 Probabilistic Ensembles with Trajectory Sampling (PETS) 63
3.4.2 Probabilistic Inference for Learning Control (PILCO) 64
3.4.3 Model Predictive Control (MPC) 65
3.4.4 Model-Agnostic Meta-Learning (MAML) 66
3.4.5 Soft Actor-Critic with Model Ensemble 67
3.4.6 Deep Deterministic Policy Gradients with Model (DDPG with Model) 68
3.5 Characteristics of Reinforcement Learning 69
3.6 DRL Algorithms and Their Advantages and Drawbacks 71
3.7 Conclusion 72
References 72
4 Deep Reinforcement Learning in Healthcare and Biomedical Applications 75
Balakrishnan D., Aarthy C., Nandhagopal Subramani, Venkatesan R. and Logesh T. R.
4.1 Introduction 76
4.2 Related Works 76
4.3 Deep Reinforcement Learning Framework 80
4.4 Deep Reinforcement Learning Applications in Healthcare and Biomedicine 81
4.5 Deep Reinforcement Learning Employs Efficient Algorithms 82
4.5.1 Deep Q-Networks 82
4.5.2 Policy Differentiation Techniques 82
4.5.3 Hindsight Experience Replay (HER) 82
4.5.4 Curiosity-Driven Exploration 82
4.5.5 Long Short-Term Memory Networks and Recurring Neural Network Designs 82
4.5.6 Multi-Agent DRL 83
4.6 Semi-Autonomous Control Based on Deep Reinforcement Learning for Robotic Surgery 83
4.6.1 Double Deep Q-Network (DDQN) 83
4.6.2 Materials and Methods 84
4.6.3 Results 86
4.6.4 Discussion 87
4.7 Conclusion 87
References 88
5 Application of Deep Reinforcement Learning in Adversarial Malware Detection 91
Manju and Chhavi Rana
5.1 Introduction 91
5.1.1 Background 95
5.1.2 Significance of Malware Detection 96
5.1.3 Challenges with Adversarial Attacks 96
5.2 Foundations of Deep Reinforcement Learning 97
5.2.1 Overview of Deep Reinforcement Learning 98
5.2.2 Core Concepts and Components 99
5.2.3 Relevance to Malware Detection 100
5.3 Malware Detection Landscape 101
5.3.1 Evolution of Malware Detection Techniques 102
5.3.2 Adversarial Attacks in Cybersecurity 103
5.3.3 Need for Advanced Detection Strategies 104
5.4 Deep Reinforcement Learning Techniques 104
5.4.1 Application of Deep Learning in Malware Detection 105
5.4.2 Reinforcement Learning Algorithms 106
5.5 Feature Selection Strategies 107
5.5.1 Importance of Feature Selection in Malware Detection 108
5.5.2 Techniques for Feature Selection 108
5.5.3 Optimization for Deep Reinforcement Learning Models 109
5.6 Datasets and Evaluation 110
5.7 Generating Adversarial Samples 111
Conclusion and Future Directions 112
Future Directions 112
References 112
6 Artificial Intelligence in Blockchain and Smart Contracts for Disruptive Innovation 115
Eashwar Sivakumar, Kiran Jot Singh and Paras Chawla
6.1 Introduction 115
6.1.1 Smart Contract 116
6.2 Literature Review 117
6.2.1 Blockchain and Smart Contracts in Digital Identity 117
6.2.2 Blockchain and Smart Contracts in Financial Security 118
6.2.3 Blockchain and Smart Contracts in Supply Chain Management 119
6.2.4 Blockchain and Smart Contracts in Insurance 120
6.2.5 Blockchain and Smart Contracts in Healthcare 121
6.2.6 Blockchain and Smart Contracts in Agriculture 121
6.2.7 Blockchain and Smart Contracts in Real Estate 122
6.2.8 Blockchain and Smart Contracts in Education and Research 123
6.2.9 Blockchain and Smart Contracts in Other Sectors 124
6.3 Critical Analysis of the Review 125
6.4 Blockchain and Artificial Intelligence 128
6.5 Discussion on the Reasoning for Implementation of Blockchain 129
6.6 Conclusion 130
References 130
7 Clinical Intelligence: Deep Reinforcement Learning for Healthcare and Biomedical Advancements 137
Keerthika K., Kannan M. and T. Saravanan
7.1 Introduction 138
7.2 Deep Reinforcement Learning Methods 138
7.2.1 Model-Free Methods 138
7.2.2 Policy Gradient Methods 139
7.2.3 Model-Based Methods 139
7.3 Applications of DRL in Healthcare 140
7.3.1 Tailored Treatment Recommendations 140
7.3.2 Optimization of Clinical Trials 141
7.3.3 Disease Diagnosis Support 142
7.3.4 Accelerated Drug Discovery and Design 142
7.3.5 Enhanced Robotic Surgery and Assistance 142
7.3.6 Health Management System 143
7.4 Challenges 143
7.5 Healthcare Data Types 144
7.5.1 Electronic Healthcare Records (EHRs) 144
7.5.2 Laboratory Data 145
7.5.3 Sensor Data 145
7.5.4 Biomedical Imaging Information 145
7.6 Guidelines for the Application of DRL 147
7.7 A Case Study: DRL in Healthcare and Biomedical Applications 147
7.7.1 Optimizing Radiation Therapy Dose Distribution in Cancer Treatment 147
7.7.2 Dose Strategy Model in Sepsis Patient Treatment 148
References 149
8 Cultivating Expertise in Deep and Reinforcement Learning Principles 151
Chilakalapudi Malathi and J. Sheela
8.1 Introduction 151
8.1.1 Reinforcement Learning's Constituent Parts 152
8.1.2 Process of Markov Decisions (MDP) 152
8.1.3 Learning Reinforcement Methods 153
8.2 Intensive Learning Foundations 164
8.2.1 A Definition of Deep Learning 164
8.2.2 Deep Learning Elements 164
8.2.2.1 Different Kinds of Deep Learning Networks 165
8.3 Integrating Deep Learning and Reinforcement Learning 172
8.3.1 Deep Reinforcement Learning 172
8.3.2 Deep Reinforcement Learning Complexity Problems 174
Conclusion 175
References 175
9 Deep Reinforcement Learning in Healthcare and Biomedical Research 179
Shruti Agrawal and Pralay Mitra
9.1 Introduction 180
9.1.1 Reinforcement Learning 180
9.1.2 Deep Reinforcement Learning 181
9.2 Learning Methods in Bioinformatics with Applications in Healthcare and Biomedical Research 182
9.2.1 Protein Folding 182
9.2.2 Protein Docking 183
9.2.3 Protein-Ligand Binding 185
9.2.4 Binding Peptide Generation 187
9.2.5 Protein Design and Engineering 188
9.2.6 Drug Discovery and Development 190
9.3 Applications in Biological Data 192
9.3.1 Omics Data 192
9.3.2 Medical Imaging 192
9.3.3 Brain/Body-Machine Interfaces 193
9.4 Adaptive Treatment Approach in Healthcare 193
9.5 Diagnostic Tools in Healthcare and Biomedical Research 195
9.6 Scope of Deep Reinforcement Learning in Healthcare and Biomedical Applications 196
9.6.1 State and Action Space 196
9.6.2 Reward 197
9.6.3 Policy 198
9.6.4 Model Training 199
9.6.5 Exploration 199
9.6.6 Credit Assignment 200
9.7 Conclusions 200
References 201
10 Deep Reinforcement Learning in Robotics and Autonomous Systems 207
Uma Yadav, Shweta V. Bondre and Bhakti Thakre
10.1 Introduction 208
10.2 The Promise of Deep Reinforcement Learning (DRL) in Real-World Robotics 210
10.3 Preliminaries 211
10.4 Enhancing RL for Real-World Robotics 222
10.5 Reinforcement Learning for Various Robotic Applications 224
10.6 Problems Faced in RL for Robotics 231
10.7 RL in Robotics: Trends and Challenges 232
10.8 Conclusion 235
References 236
11 Diabetic Retinopathy Detection and Classification Using Deep Reinforcement Learning 239
H.R. Manjunatha and P. Sathish
11.1 Introduction 239
11.2 Literature Survey 243
11.3 Diabetic Retinopathy Detection and Classification 248
11.4 Result Analysis 256
11.5 Conclusion 260
References 260
12 Early Brain Stroke Detection Based on Optimized Cuckoo Search Using LSTM-Gated Multi-Perceptron Neural Network 265
Anita Venaik, Asha A., Dhiyanesh B., Kiruthiga G., Shakkeera L. and Vinodkumar Jacob
12.1 Introduction 266
12.2 Literature Survey 268
12.2.1 Problem Statement 269
12.3 Proposed Methodology 270
12.3.1 Dataset Collection 270
12.3.2 Preprocessing 271
12.3.3 Genetic Feature Sequence Algorithm (GFSA) 275
12.3.4 Disease-Prone Factor (DPF) 281
12.3.5 Decision Tree-Optimized Cuckoo Search (DTOCS) 284
12.3.6 Long Short-Term Memory Gate Multilayer Perceptron Neural Network (LSTM-MLPNN) 289
12.4 Result and Discussion 293
12.4.1 Performance Matrix 293
12.5 Conclusion 296
References 297
13 Hybrid Approaches: Combining Deep Reinforcement Learning with Other Techniques 301
M. T. Vasumathi, Manju Sadasivan and Aurangjeb Khan
13.1 Introduction 302
13.1.1 Digital Twin-Introduction 302
13.1.2 Model of a Digital Twin 302
13.1.2.1 Steps Involved in Building a Digital Twin Prototype 303
13.1.3 Application Areas of Digital Twins 303
13.1.3.1 Digital Twin in Medical Field 304
13.1.3.2 Digital Twin in Smart City 304
13.1.3.3 Digital Twin in Sports 304
13.1.3.4 Digital Twin in Smart Manufacturing 305
13.2 Digital Twin Technologies 305
13.2.1 Data Acquisition and Sensors 306
13.2.2 Data Analytics and Machine Learning 306
13.2.3 Cloud Computing 307
13.2.4 Other Technologies 307
13.3 Integration of RL and Digital Twin 307
13.3.1 Motivation for Combining Digital Twin and RL 309
13.3.2 How RL Enhances Decision-Making Within Digital Twins 310
13.4 Challenges of Using RL in Digital Twins 311
13.5 Digital Twin Modeling with RL 312
13.6 Technology Underlying RL-Based Digital Twins 314
13.6.1 Integration of RL with Digital Twins in Four Stages 314
13.6.2 Tools and Libraries for Developing RL-Based Digital Twins 314
13.6.2.1 Simulation and Digital Twin Platforms 314
13.6.2.2 Reinforcement Learning Libraries 315
13.6.3 Integration with Existing Systems and IoT Devices for RL Deployment 315
13.6.3.1 Data Collection and Sensor Integration 315
13.6.3.2 Communication and Data Ingestion 316
13.6.3.3 Digital Twin Integration 316
13.6.3.4 RL Integration 316
13.6.3.5 Control and Actuation 316
13.6.3.6 Implementation of Feedback and Learning Process 316
13.6.3.7 Dashboard for Alert and Visualization 316
13.6.3.8 Ensuring the Security and Authentication 317
13.7 Industry-Specific Applications: A Case Study of DT in a Car Manufacturing Unit 317
13.7.1 IoT Components Required for Creating Digital Twin for the Manufacturing Unit 318
13.7.2 Architecture of the Proposed Digital Twin for Car Manufacturing Unit 318
13.7.3 Challenges and Opportunities in the Implementation of DTs for Car Manufacturing 320
13.8 Conclusion 321
References 322
14 Predictive Modeling of Rheumatoid Arthritis Symptoms: A High-Performance Approach Using HSFO-SVM and UNET-CNN 325
Anusuya V., Baseera A., Dhiyanesh B., Parveen Begam Abdul Kareem and Shanmugaraja P.
14.1 Introduction 326
14.1.1 Novelty of the Research 327
14.2 Related Work 328
14.2.1 Challenges and Problem Identification Factor 331
14.3 HSFO-SVM Based on LSTM-Gated Convolution Neural Network (lstmg-cnn) 332
14.3.1 C-Score and Cross-Fold Validation 332
14.3.2 Honey Scout Forager Optimization 335
14.3.3 Feature Selection Using SVM 336
14.3.4 UNET-CNN Classification 338
14.4 Result and Discussion 341
14.5 Conclusion 345
References 346
15 Using Reinforcement Learning in Unity Environments for Training AI Agent 349
Geetika Munjal and Monika Lamba
15.1 Introduction 349
15.2 Literature Review 351
15.3 Machine Learning 352
15.3.1 Categorization of Machine Learning 352
15.3.1.1 Supervised Learning 352
15.3.1.2 Unsupervised Learning 353
15.3.1.3 Reinforcement Learning 353
15.3.2 Classifying on the Basis of Envisioned Output 353
15.3.2.1 Classification 354
15.3.2.2 Regression 354
15.3.2.3 Clustering 354
15.3.3 Artificial Intelligence 354
15.4 Unity 354
15.4.1 Unity Hub 355
15.4.2 Unity Editor 355
15.4.3 Inspector 355
15.4.4 Game View 355
15.4.5 Scene View 355
15.4.6 Hierarchy 355
15.4.7 Project Window 356
15.5 Reinforcement Learning and Supervised Learning 356
15.5.1 Positive Reinforcement 357
15.5.2 Negative Reinforcement 357
15.5.3 Model-Free and Model-Based RL 357
15.6 Proposed Model 359
15.6.1 Setting Up a Virtual Environment 359
15.6.2 Setting Up of the Environment 360
15.6.2.1 Creating and Allocating Scripts for the Environment 361
15.6.2.2 Creating a Goal for the Agent 361
15.6.2.3 Reward-Driven Behavior 361
15.7 Markov Decision Process 362
15.8 Model-Based RL 362
15.9 Experimental Results 363
15.9.1 Machine Learning Models Used for the Environments 363
15.9.2 PushBlock 363
15.9.3 Hallway 365
15.9.4 Screenshots of the PushBlock Environment 368
15.9.5 Screenshots of the Hallway Environment 369
15.10 Conclusion 372
References 372
16 Emerging Technologies in Healthcare Systems 375
Ravi Kumar Sachdeva, Priyanka Bathla, Samriti Vij, Dishika, Madhur Jain, Lokesh Kumar, G. S. Pradeep Ghantasala and Rakesh Ahuja
16.1 Introduction 375
16.2 Personalized Medicine 376
16.3 AI and ML in Healthcare Sector 377
16.3.1 AI in Medical Diagnosis 378
16.3.2 Drug Discovery 378
16.3.3 Personalized Treatment Plans 379
16.3.4 Pattern Matching or Trend Detection 380
16.4 Immunotherapy 380
16.4.1 Monoclonal Antibodies 381
16.4.2 Checkpoint Inhibitors 381
16.4.3 CAR-T Cell Therapy 381
16.5 Regenerative Medicine 381
16.6 Digital Health (Use of Technology in Healthcare) 383
16.6.1 Wearable Devices 383
16.6.2 Telemedicine 384
16.6.3 Electronic Health Records 384
16.7 Health Inequity 385
16.7.1 Health Disparity 385
16.7.2 Health Equity 385
16.8 Future Directions in Healthcare Research 385
16.9 Challenges and Recommendations for Advanced Level of Modern Healthcare Technologies 386
16.9.1 Challenges 387
16.9.2 Recommendations 388
16.10 Healthcare Sector in Developing and Underdeveloped Countries 388
16.10.1 Healthcare Sector in Developing Countries 388
16.10.2 Healthcare Sector in Underdeveloped Countries 389
16.11 Comparison of Recent Progress and Future Mentoring in Healthcare Using Technology 389
16.12 Conclusion 391
References 392
Index 395