SAP Data Intelligence
The Comprehensive Guide
1st Edition
Published on 24. November 2021
Book
Hardback
783 pages
978-1-4932-2162-2 (ISBN)
Description
Manage your data landscape with SAP Data Intelligence! Begin by understanding its architecture and capabilities and then see how to set up and install SAP Data Intelligence with step-by-step instructions. Walk through SAP Data Intelligence applications and learn how to use them for data governance, orchestration, and machine learning. Integrate with ABAP-based systems, SAP Vora, SAP Analytics Cloud, and more. Manage, secure, and operate SAP Data Intelligence with this all-in-one guide!
In this book, you'll learn about:
a. Configuration
Build your SAP Data Intelligence landscape! Use SAP Cloud Appliance Library for cloud deployment, including provisioning, sizing, and accessing the launchpad. Perform on-premise installations using tools like the maintenance planner.
b. Capabilities
Put the core capabilities of SAP Data Intelligence to work! Manage and govern your data with the metadata explorer, use the modeler application to create data processing pipelines, create apps with the Jupyter Notebook, and more.
c. Integration and Administration
Integrate, manage, and operate SAP Data Intelligence! Get step-by-step instructions for integration with SAP and non-SAP systems. Learn about key administration tasks and make sure your landscape is secure and running smoothly.
Highlights include:
1) Configuration and installation
2) Data governance
3) Data processing pipelines
4) Docker images
5) ML Scenario Manager
6) Jupyter Notebook
7) Python SDK
8) Integration
9) Administration
10) Security
11) Application lifecycle management
12) Use cases
More details
Persons
Author
Dharma Teja Atluri is an executive architect and artificial intelligence/machine learning evangelist at IBM. He has more than 18 years of experience working in advanced analytics with both SAP and non-SAP product lines. He has provided strategic direction to clients globally regarding the adoption of SAP and non-SAP advanced analytics products for artificial intelligence/machine learning operationalization, data management, information management, and analytics. He has also carried out multiple platform comparison initiatives for reporting, extract, transform load (ETL), data warehousing, and data science products across IBM, Microsoft Azure, Google, Amazon Web Services, and SAP. He has led the SAP analytics (reporting and enterprise information management) portfolio for IBM India, and designed client architectures for analytics with SAP and IBM capabilities. Dharma is an IBM master certified data scientist, architect, and technical specialist, and also an IBM thought leader certified consultant. His most recent SAP Data Intelligence sprint was featured for global consumption by clients and nominated for SAP Innovation Awards. He can be reached at https://www.linkedin.com/in/dharma.
ISNI:
ISNI:
Devraj Bardhan is a distinguished global SAP leader at IBM, renowned for his exceptional track record in leading large-scale transformation projects that have significantly driven business growth through innovation and digital efficiencies. As an esteemed subject matter expert (SME) in SAP S/4HANA, SAP Business Technology Platform (SAP BTP), and artificial intelligence (AI), Devraj has played a pivotal role in expanding IBM's digital transformation capabilities by developing and implementing global templates and intelligent workflows. He maintains strong collaborative relationships with both SAP and IBM customers, fostering the incubation of groundbreaking ideas. Additionally, Devraj spearheads the NextLevel SAP BTP & AI partnership, which leverages joint intellectual property creation between IBM and SAP to drive mutual success.Devraj can be reached via www.linkedin.com/in/bardhan or on social media channels like X, Bluesky, and Mastodon as @devbard.
ISNI:
ISNI:
Santanu Ghosh is an SAP analytics practitioner working as a consultant for the past 15 years in the data warehouse space. He has worked with SAP Business Warehouse, SAP HANA, SAP BusinessObjects BI, and SAP Analytics Cloud. Recently, he has been working with the SAP Data Intelligence platform, delivering customer use cases and business content for the automotive industry. Santanu is a data science and machine learning enthusiast, and is pursuing a degree in data science. He has worked on multiple predictive analytics and machine learning projects over the past four years, for a variety of industries, leveraging SAP Data Intelligence, SAP Predictive Analytics, SAP HANA Predictive Analysis Library, and SAP HANA Automated Predictive Library. He is also experienced in programming using Python and R languages.
Snehasish Ghosh is an enterprise information management consultant at IBM with more than 13 years of experience working with SAP's analytics and information management product portfolio. He has extensive experience in data integration and migration, data quality, and reporting. He has implemented a variety of SAP and non-SAP solutions, including SAP S/4HANA, SAP Business Warehouse, Amazon Web Services, and Salesforce. He has also worked with SAP enterprise information management solutions, including SAP Data Intelligence, SAP Information Steward, SAP Data Services, and SAP HANA smart data integration and smart data quality.
Arindom Saha is an SAP business intelligence consultant with more than 11 years of experience working with the SAP analytics portfolio. He has extensive experience in SAP and non-SAP reporting and visualization products. He has implemented a variety of analytics solutions on different products, including SAP BusinessObjects BI, SAP HANA, SAP Business Warehouse, and Oracle RDBMS. He also has experience working with SAP Lumira, SAP Analytics Cloud, SAP Data Intelligence, and Tableau. His recent experience with SAP Data Intelligence has been contributing business content and working closely with cloud and on-premise providers to install and provision SAP Data Intelligence 3.1 as part of ongoing global initiatives.
Content
... Preface ... 21
... Why Read This Book? ... 21
... Audience ... 22
... Structure of the Book ... 23
... Acknowledgments ... 28
... Conclusion ... 29
PART I ... Getting Started ... 31
1 ... The Data Fabric for the Intelligent Enterprise ... 33
1.1 ... Data Fabric ... 34
1.2 ... Data Orchestration ... 38
1.3 ... SAP Business Technology Platform ... 40
1.4 ... SAP Data Intelligence ... 43
1.5 ... Summary ... 50
2 ... Architecture and Capabilities ... 51
2.1 ... Genesis of SAP Data Intelligence ... 52
2.2 ... SAP Data Intelligence Architecture ... 60
2.3 ... Deployment Options and Bring Your Own License Model ... 63
2.4 ... Kubernetes Cluster and Containers ... 68
2.5 ... SAP Data Intelligence Launchpad ... 86
2.6 ... Summary ... 91
3 ... Setup and Installation ... 93
3.1 ... Landscape Sizing ... 93
3.2 ... SAP Cloud Appliance Library ... 99
3.3 ... On-Demand Cloud Provisioning and Instance Sizing ... 107
3.4 ... Setting Up SAP Data Intelligence on SAP Cloud Appliance Library ... 113
3.5 ... SAP Data Intelligence 3.0 Installation On-Premise ... 150
3.6 ... Summary ... 168
4 ... Using SAP Data Intelligence Applications ... 169
4.1 ... SAP Data Intelligence Launchpad Applications ... 169
4.2 ... Applications for Data Engineers ... 172
4.3 ... Applications for Data Scientists ... 177
4.4 ... Applications for Modelers and Auditors ... 179
4.5 ... Applications for System Administrators ... 182
4.6 ... Summary ... 189
PART II ... Data Management, Orchestration, and Machine Learning ... 191
5 ... Metadata-Driven Data Governance ... 193
5.1 ... Metadata Explorer for Data Governance ... 194
5.2 ... Data Profiling to Understand Data ... 197
5.3 ... Managing Publications and Data Catalogs ... 202
5.4 ... Defining Data Quality Rules and Running Rulebooks ... 214
5.5 ... Data Lineage from Transformation History ... 230
5.6 ... Summary ... 235
6 ... Modeling Data Processing Pipelines ... 237
6.1 ... Using the SAP Data Intelligence Modeler ... 237
6.2 ... Creating and Managing Connections ... 250
6.3 ... Self-Service Data Preparation with the Metadata Explorer ... 255
6.4 ... Integrating, Processing, and Orchestrating Workflows ... 261
6.5 ... Scheduling and Monitoring Data Pipelines ... 270
6.6 ... Summary ... 273
7 ... Creating Operators and Data Types ... 275
7.1 ... Creating Custom Operators ... 276
7.2 ... Implementing Runtime Operators ... 288
7.3 ... Creating Data Types ... 290
7.4 ... Summary ... 293
8 ... Building Docker Images ... 295
8.1 ... Containers in Pods and Pods in Clusters ... 295
8.2 ... Assembling a Docker Image ... 298
8.3 ... Dockerfile Inheritance ... 303
8.4 ... Using Docker with Python ... 305
8.5 ... Summary ... 308
9 ... Machine Learning ... 309
9.1 ... Machine Learning with SAP ... 310
9.2 ... Machine Learning with SAP Data Intelligence ... 328
9.3 ... Using the ML Scenario Manager ... 333
9.4 ... ML Data Manager in Data Workspaces and Data Collections ... 365
9.5 ... Summary ... 371
10 ... Jupyter Notebook ... 373
10.1 ... Jupyter Notebook Fundamentals ... 374
10.2 ... Working with SAP HANA Cloud ... 386
10.3 ... Data Science Experiments with Jupyter Notebook ... 405
10.4 ... JupyterLab as the Next-Gen Jupyter Notebook ... 430
10.5 ... Summary ... 437
11 ... SAP Data Intelligence Python SDK ... 439
11.1 ... Using SAP Data Intelligence Python SDK ... 440
11.2 ... Accessing Artifacts Using Methods ... 448
11.3 ... Machine Learning Tracking SDK ... 450
11.4 ... Summary ... 454
PART III ... Integration ... 457
12 ... Integrating with ABAP Systems ... 459
12.1 ... Integration Scenarios ... 459
12.2 ... Provisioning Data from ABAP Systems ... 465
12.3 ... Using Operators to Trigger Execution in an ABAP System ... 472
12.4 ... SAP BW/4HANA and SAP Data Intelligence Hybrid Data Virtualization ... 478
12.5 ... Additional Connectivity ... 485
12.6 ... Summary ... 495
13 ... Integrating with Non-SAP Systems ... 497
13.1 ... Non-SAP Cloud System Connectivity ... 497
13.2 ... Non-SAP On-Premise System Connectivity ... 510
13.3 ... Summary ... 513
14 ... Integrating Big Data Workloads with SAP Vora ... 515
14.1 ... SAP Vora in Kubernetes Framework ... 516
14.2 ... Data Modeling in SAP Vora ... 524
14.3 ... Hierarchies in SAP Vora ... 536
14.4 ... Full-Text Search in SAP Vora ... 540
14.5 ... Summary ... 542
15 ... Integrating with SAP Data Warehouse Cloud ... 543
15.1 ... Overview of SAP Data Warehouse Cloud ... 543
15.2 ... Understanding Spaces ... 549
15.3 ... Exploring Connections and Using the Data Builder ... 561
15.4 ... Data Builder in SAP Data Warehouse Cloud versus Pipelines in SAP Data Intelligence ... 570
15.5 ... Summary ... 570
16 ... Integrating with SAP Analytics Cloud ... 571
16.1 ... Overview of SAP Analytics Cloud ... 571
16.2 ... Use Operators: Read File, Formatter, and Producer ... 582
16.3 ... Pipelines to Train, Predict, and Visualize Data ... 587
16.4 ... Summary ... 591
PART IV ... System Management, Security, and Operations ... 593
17 ... Administration ... 595
17.1 ... System Management Command-Line Client Reference ... 595
17.2 ... Administration Applications ... 599
17.3 ... Monitoring the SAP Data Intelligence Modeler ... 616
17.4 ... SAP Data Intelligence System Logging ... 626
17.5 ... System Diagnostics ... 631
17.6 ... Summary ... 637
18 ... Security ... 639
18.1 ... Approach to Data Protection ... 639
18.2 ... Authenticating Services and Users ... 642
18.3 ... Securely Connecting On-Premise Systems ... 658
18.4 ... Summary ... 659
19 ... Maintenance ... 661
19.1 ... Understanding Operational Modes or Run Levels ... 661
19.2 ... Switching the Platform to Maintenance Mode ... 662
19.3 ... Increasing System Management Persistent Volume Size ... 665
19.4 ... Performing Backups ... 668
19.5 ... Summary ... 671
20 ... Application Lifecycle Management ... 673
20.1 ... Version Control System ... 673
20.2 ... Git ... 674
20.3 ... Continuous Integration and Continuous Delivery ... 707
20.4 ... DevOps Fundamentals and Tools ... 713
20.5 ... SAP Data Intelligence as the MLOps Platform ... 723
20.6 ... Migrating from SAP Leonardo Machine Learning Foundation ... 730
20.7 ... Summary ... 734
21 ... Business Content and Use Cases ... 737
21.1 ... Digital Transformation and SAP Data Intelligence ... 737
21.2 ... Business Content by Industry ... 740
21.3 ... Finance Use Cases ... 746
21.4 ... Supply Chain Use Cases ... 747
21.5 ... Manufacturing Use Cases ... 749
21.6 ... Summary ... 751
... Appendices ... 753
A ... Outlook and Roadmap ... 753
B ... The Authors ... 763
... Index ... 765
... Why Read This Book? ... 21
... Audience ... 22
... Structure of the Book ... 23
... Acknowledgments ... 28
... Conclusion ... 29
PART I ... Getting Started ... 31
1 ... The Data Fabric for the Intelligent Enterprise ... 33
1.1 ... Data Fabric ... 34
1.2 ... Data Orchestration ... 38
1.3 ... SAP Business Technology Platform ... 40
1.4 ... SAP Data Intelligence ... 43
1.5 ... Summary ... 50
2 ... Architecture and Capabilities ... 51
2.1 ... Genesis of SAP Data Intelligence ... 52
2.2 ... SAP Data Intelligence Architecture ... 60
2.3 ... Deployment Options and Bring Your Own License Model ... 63
2.4 ... Kubernetes Cluster and Containers ... 68
2.5 ... SAP Data Intelligence Launchpad ... 86
2.6 ... Summary ... 91
3 ... Setup and Installation ... 93
3.1 ... Landscape Sizing ... 93
3.2 ... SAP Cloud Appliance Library ... 99
3.3 ... On-Demand Cloud Provisioning and Instance Sizing ... 107
3.4 ... Setting Up SAP Data Intelligence on SAP Cloud Appliance Library ... 113
3.5 ... SAP Data Intelligence 3.0 Installation On-Premise ... 150
3.6 ... Summary ... 168
4 ... Using SAP Data Intelligence Applications ... 169
4.1 ... SAP Data Intelligence Launchpad Applications ... 169
4.2 ... Applications for Data Engineers ... 172
4.3 ... Applications for Data Scientists ... 177
4.4 ... Applications for Modelers and Auditors ... 179
4.5 ... Applications for System Administrators ... 182
4.6 ... Summary ... 189
PART II ... Data Management, Orchestration, and Machine Learning ... 191
5 ... Metadata-Driven Data Governance ... 193
5.1 ... Metadata Explorer for Data Governance ... 194
5.2 ... Data Profiling to Understand Data ... 197
5.3 ... Managing Publications and Data Catalogs ... 202
5.4 ... Defining Data Quality Rules and Running Rulebooks ... 214
5.5 ... Data Lineage from Transformation History ... 230
5.6 ... Summary ... 235
6 ... Modeling Data Processing Pipelines ... 237
6.1 ... Using the SAP Data Intelligence Modeler ... 237
6.2 ... Creating and Managing Connections ... 250
6.3 ... Self-Service Data Preparation with the Metadata Explorer ... 255
6.4 ... Integrating, Processing, and Orchestrating Workflows ... 261
6.5 ... Scheduling and Monitoring Data Pipelines ... 270
6.6 ... Summary ... 273
7 ... Creating Operators and Data Types ... 275
7.1 ... Creating Custom Operators ... 276
7.2 ... Implementing Runtime Operators ... 288
7.3 ... Creating Data Types ... 290
7.4 ... Summary ... 293
8 ... Building Docker Images ... 295
8.1 ... Containers in Pods and Pods in Clusters ... 295
8.2 ... Assembling a Docker Image ... 298
8.3 ... Dockerfile Inheritance ... 303
8.4 ... Using Docker with Python ... 305
8.5 ... Summary ... 308
9 ... Machine Learning ... 309
9.1 ... Machine Learning with SAP ... 310
9.2 ... Machine Learning with SAP Data Intelligence ... 328
9.3 ... Using the ML Scenario Manager ... 333
9.4 ... ML Data Manager in Data Workspaces and Data Collections ... 365
9.5 ... Summary ... 371
10 ... Jupyter Notebook ... 373
10.1 ... Jupyter Notebook Fundamentals ... 374
10.2 ... Working with SAP HANA Cloud ... 386
10.3 ... Data Science Experiments with Jupyter Notebook ... 405
10.4 ... JupyterLab as the Next-Gen Jupyter Notebook ... 430
10.5 ... Summary ... 437
11 ... SAP Data Intelligence Python SDK ... 439
11.1 ... Using SAP Data Intelligence Python SDK ... 440
11.2 ... Accessing Artifacts Using Methods ... 448
11.3 ... Machine Learning Tracking SDK ... 450
11.4 ... Summary ... 454
PART III ... Integration ... 457
12 ... Integrating with ABAP Systems ... 459
12.1 ... Integration Scenarios ... 459
12.2 ... Provisioning Data from ABAP Systems ... 465
12.3 ... Using Operators to Trigger Execution in an ABAP System ... 472
12.4 ... SAP BW/4HANA and SAP Data Intelligence Hybrid Data Virtualization ... 478
12.5 ... Additional Connectivity ... 485
12.6 ... Summary ... 495
13 ... Integrating with Non-SAP Systems ... 497
13.1 ... Non-SAP Cloud System Connectivity ... 497
13.2 ... Non-SAP On-Premise System Connectivity ... 510
13.3 ... Summary ... 513
14 ... Integrating Big Data Workloads with SAP Vora ... 515
14.1 ... SAP Vora in Kubernetes Framework ... 516
14.2 ... Data Modeling in SAP Vora ... 524
14.3 ... Hierarchies in SAP Vora ... 536
14.4 ... Full-Text Search in SAP Vora ... 540
14.5 ... Summary ... 542
15 ... Integrating with SAP Data Warehouse Cloud ... 543
15.1 ... Overview of SAP Data Warehouse Cloud ... 543
15.2 ... Understanding Spaces ... 549
15.3 ... Exploring Connections and Using the Data Builder ... 561
15.4 ... Data Builder in SAP Data Warehouse Cloud versus Pipelines in SAP Data Intelligence ... 570
15.5 ... Summary ... 570
16 ... Integrating with SAP Analytics Cloud ... 571
16.1 ... Overview of SAP Analytics Cloud ... 571
16.2 ... Use Operators: Read File, Formatter, and Producer ... 582
16.3 ... Pipelines to Train, Predict, and Visualize Data ... 587
16.4 ... Summary ... 591
PART IV ... System Management, Security, and Operations ... 593
17 ... Administration ... 595
17.1 ... System Management Command-Line Client Reference ... 595
17.2 ... Administration Applications ... 599
17.3 ... Monitoring the SAP Data Intelligence Modeler ... 616
17.4 ... SAP Data Intelligence System Logging ... 626
17.5 ... System Diagnostics ... 631
17.6 ... Summary ... 637
18 ... Security ... 639
18.1 ... Approach to Data Protection ... 639
18.2 ... Authenticating Services and Users ... 642
18.3 ... Securely Connecting On-Premise Systems ... 658
18.4 ... Summary ... 659
19 ... Maintenance ... 661
19.1 ... Understanding Operational Modes or Run Levels ... 661
19.2 ... Switching the Platform to Maintenance Mode ... 662
19.3 ... Increasing System Management Persistent Volume Size ... 665
19.4 ... Performing Backups ... 668
19.5 ... Summary ... 671
20 ... Application Lifecycle Management ... 673
20.1 ... Version Control System ... 673
20.2 ... Git ... 674
20.3 ... Continuous Integration and Continuous Delivery ... 707
20.4 ... DevOps Fundamentals and Tools ... 713
20.5 ... SAP Data Intelligence as the MLOps Platform ... 723
20.6 ... Migrating from SAP Leonardo Machine Learning Foundation ... 730
20.7 ... Summary ... 734
21 ... Business Content and Use Cases ... 737
21.1 ... Digital Transformation and SAP Data Intelligence ... 737
21.2 ... Business Content by Industry ... 740
21.3 ... Finance Use Cases ... 746
21.4 ... Supply Chain Use Cases ... 747
21.5 ... Manufacturing Use Cases ... 749
21.6 ... Summary ... 751
... Appendices ... 753
A ... Outlook and Roadmap ... 753
B ... The Authors ... 763
... Index ... 765