The AI Product Playbook
Description
A comprehensive guide for aspiring and current AI product managers
The AI Product Playbook: Strategies, Skills, and Frameworks for the AI-Driven Product Manager, by Dr. Marily Nika and Diego Granados, is a practical resource designed to empower product managers to effectively build, launch, and manage successful AI-powered products. This playbook bridges the gap between artificial intelligence theory and real-world product management, offering actionable learnings tailored to non-technical professionals.
Drawing from extensive industry experience, Dr. Nika and Granados introduce the three essential AI product manager roles: AI Experiences PM, AI Builder PM, and AI-Enhanced PM. They offer guidance on developing skills crucial for each role and navigating common challenges in the workplace. Readers will also find valuable strategies for career growth, lifelong learning, and crafting a distinctive AI portfolio.
Inside the book:
- Practical frameworks for discovering AI opportunities and aligning AI capabilities with business goals
- A deep technical dive with clear explanations of foundational AI and machine learning concepts, including supervised learning, unsupervised learning, reinforcement learning, and generative AI
- Guidelines for ethical AI implementation, addressing bias, fairness, and compliance with AI regulations
- Strategies for effective collaboration with cross-functional teams and enhancing productivity through AI
- Interactive exercises, action plans, checklists, templates, and quizzes designed to reinforce learning and build real-world skills
Essential reading for aspiring and experienced product managers alike, The AI Product Playbook provides a roadmap to mastering AI-driven product management and advancing your career in the dynamic field of artificial intelligence.
More details
Other editions
Additional editions
Persons
Dr. Marily Nika is an award-winning GenAI Product Leader at Google and one of the world's foremost AI educators, with over 13 years of experience building AI products at Google and Meta. She holds a PhD in machine learning and is an author, TED AI speaker, Harvard Business School fellow and co-founder of the AI Product Hub (www.aiproduct.com) which offers AI product management certifications.
Diego Granados is a Product Leader with more than 6 years of experience bringing AI products to life in top tech companies in Silicon Valley. He holds an MBA from Duke University and an M.S. in C.S. focused on AI & ML from Georgia Tech and is co-founder of the AI Product Hub (www.aiproduct.com) which offers AI product management certifications.
Content
Introduction xix
Part I Foundational AI/ML Concepts 1
Chapter 1 Artificial Intelligence and Machine Learning: What Every Product Manager Needs to Know 3
AI vs. ml 4
Why This Matters to a PM 4
Key Differences Between AI and ml 5
Common Misconceptions for PMs: Myths vs. Reality 7
Your Glossary as a PM 7
Grounding the Concepts: Real-World AI in Action 10
The AI PM's Guiding Principles 14
Chapter Summary and Key Takeaways 16
Key Takeaways 16
Onward: Peeking Under the Hood 17
Chapter 2 How Machine Learning Models Learn: A Peek Under the Hood 19
The Learning Process: Training, Validation, and Testing 20
How Models Learn: An Example with k-Nearest Neighbors (k-NN) 22
Applying k-NN (with k=1): 23
Another Example: Testing an Unknown Fruit 26
Evaluating Model Performance 27
The Confusion Matrix: A Foundation for Understanding 27
Key Classification Metrics (and Their PM Implications) 28
The Precision-Recall Trade-Off 29
Choosing the Right Metric 30
Overfitting and Underfitting: Striking the Right Balance for Real-World Performance 31
Overfitting: Memorizing Instead of Learning 31
Underfitting: Missing the Forest for the Trees 32
Visual Analogy: Fitting a Curve 32
Finding the Sweet Spot: Generalization 33
The PM's Role 33
Human-in-the-Loop: Blending AI Power with Human Expertise 34
What Is Human-in-the-Loop? 34
Why HITL Is Essential for Product Managers (and Their Products) 35
How to Implement HITL (PM Considerations) 37
Chapter Summary and Key Takeaways 38
Key Takeaways 39
Onward: Understanding the Broader Process 39
Chapter 3 The Big Picture: AI, ML, and You 41
Understanding the Relationship Between AI, ML, and Product Goals 41
Types of Machine Learning: Understanding the Spectrum of Learning 44
Supervised Learning: Guiding the Model with Labeled Examples 46
Technical Deep Dive: How Supervised Learning Models Learn from Labeled Data 48
Critical Considerations for Product Managers 54
Unsupervised Learning: Discovering Hidden Patterns in Your Data 55
Technical Deep Dive: How Unsupervised Learning Models Discover Patterns 57
Critical Considerations for Product Managers 60
Reinforcement Learning: Learning Through Trial and Error 61
Technical Deep Dive: How Reinforcement Learning Agents Learn Optimal Policies 63
The Learning Process: Exploration, Exploitation, and Q-Learning 65
Critical Considerations for Product Managers 67
Generative AI: Powering a New Era of Language-Based Applications 67
Technical Deep Dive: How LLMs Understand and Generate Language 69
Critical Considerations for Product Managers 72
The "Gotchas": A PM's Guide to LLM Limitations and Risks 73
Navigating the Nuances of Generative AI: Understanding GenAI Evaluations- Ensuring Quality and Trust 75
Prompt Engineering: The Art and Science of Talking to AI 84
Types of Machine Learning: A Recap 89
Introduction to Neural Networks and Deep Learning: The Engines of Complex Pattern Recognition 92
Neural Networks: Mimicking the Brain's Connections (But Not Really) 92
How Neural Networks Learn: Adjusting the Connections 94
Technical Deep Dive: The Mechanics of Neural Networks and Deep Learning 95
Challenges in Deep Learning 98
Chapter Summary and Key Takeaways 99
Key Takeaways 99
Onward: Mapping the Process 100
Chapter 4 The AI Lifecycle 101
Problem Definition and Business Understanding: The "Why" 102
Data Collection and Exploration: Understanding Your Ingredients 103
Data Preprocessing: Preparing the Ingredients 104
Feature Engineering: Crafting the Inputs for Success 104
Model Selection and Training: Choosing the Right Algorithm 105
Model Evaluation and Tuning: Ensuring Quality 106
Model Deployment and Monitoring: Bringing AI to Life (and Keeping It Healthy) 107
Retraining and Maintenance: Keeping Your Model Up-to-Date 108
Chapter Summary and Key Takeaways 109
Key Takeaways 109
Onward: Exploring the AI PM Roles 110
Part II AI PM Specializations 111
Chapter 5 AI-Experiences PM: Shaping User Interaction with AI 113
Key Responsibilities: Shaping the AI User Experience 114
Day-to-Day Activities 117
Required Skills and Knowledge: The AI-Experiences PM Toolkit 120
Core Product Management Craft and Practices 120
Engineering Foundations for PMs 121
Essential Leadership and Collaboration Skills 122
AI Lifecycle and Operational Awareness 123
Illustrative Example: A Day in the Life of an AI-Experiences PM 124
Challenges and Complexities 127
How the AI-Experiences PM Interacts with Other Roles 129
Chapter Summary and Key Takeaways 134
Key Takeaways 134
Onward: Architecting the AI Foundation 135
Chapter 6 AI-Builder PM: Architecting the Foundation of Intelligent Systems 137
Key Responsibilities: Building and Managing the AI Foundation 138
Day-to-Day Activities 141
Required Skills and Knowledge: The AI-Builder PM's Technical and Strategic Toolkit 144
Core Product Management Craft and Practices 145
Engineering Foundations for PMs 146
Essential Leadership and Collaboration Skills 147
AI Lifecycle and Operational Awareness 148
Illustrative Example: A Day in the Life of an AI-Builder PM 149
Challenges and Complexities 152
How the AI-Builder PM Interacts with Other Roles 154
Chapter Summary and Key Takeaways 156
Key Takeaways 157
Onward: Supercharging the PM Workflow 158
Chapter 7 AI-Enhanced PM: Supercharging Product Management with AI 159
Key Responsibilities: Augmenting PM Workflows and Decision-Making with AI 160
Day-to-Day Activities 162
Required Skills and Knowledge: The AI-Enhanced PM's Toolkit 165
Core Product Management Craft and Practices 165
Engineering Foundations for PMs 166
Essential Leadership and Collaboration Skills 167
AI Lifecycle and Operational Awareness 168
Illustrative Example: A Day in the Life of an AI-Enhanced PM 169
Examples of AI Tools 172
Challenges and Complexities 173
How the AI-Enhanced PM Interacts with Other Roles 175
Skill Comparison: AI-Experiences PM, AI-Builder PM, and AI-Enhanced PM 177
Chapter Summary and Key Takeaways 184
Key Takeaways 185
Onward: From Theory to Action 185
Part III Connecting the Dots Between AI/ML Knowledge and PM Craft 187
Chapter 8 Identifying and Evaluating AI Opportunities 189
Uncovering Potential Use Cases-Mining Your Product for AI Gold 189
Recognizing Data-Rich Problem Areas 190
Analyzing Existing Data Sources 192
Asking the Right Questions 193
AI/ML Capability Matching: Connecting Problems to Solutions 194
Understanding Your AI/ML Toolkit: Key Capabilities 195
Matching Capabilities to Problems: A Practical Approach 200
Feature: Search Functionality in a Document Management System 200
Feature: Customer Support Chatbot 201
Feature: Reporting Dashboard for Marketing Campaigns 201
Finding AI Opportunities in the User Journey 202
Mapping the User Journey: Charting the Course 202
Identifying Pain Points and Opportunities: The AI Detective Work 204
Applying AI/ML to Enhance Touchpoints: The Transformation 205
Feature Enhancement Through AI/ML- Transforming Existing Functionality 208
Identifying Enhancement Opportunities: Finding the Weak Spots 209
Applying AI/ML to Enhance Features: The Transformation Process 210
Feature: Standard Search Functionality 212
Feature: Data Entry Form 212
Feature: Reporting Dashboard 212
Proactive Product Management-Anticipating User Needs with AI 213
Understanding the Power of Prediction and Automation 213
Key Areas for Predictive and Automation Opportunities 214
Identifying Opportunities: A Practical Approach 216
Responsible AI Foundations-Ethical and Feasibility Considerations 217
Ethical Considerations: The "Do No Harm" Principle 217
Feasibility Considerations: Can We Actually Build This? 220
Practical Ideation Techniques for AI/ML Use Cases-Thinking Like an AI-First Product Manager 221
Ideation Techniques: Unleashing Your AI Creativity 222
"AI Feature Storming": The Brain Dump 222
"AI Scenario Planning": Walking in the User's Shoes 223
"Data Opportunity Mapping": Leveraging Your Data Assets 223
"AI Capability Alignment": The Matching Game 224
"AI-Powered Feature Reverse Engineering": Learning from Others 225
Cultivating an AI-First Mindset 226
Chapter Summary and Key Takeaways 226
Key Takeaways 227
Onward: Measuring the Value of Your Ideas 227
Chapter 9 ROI Calculation for AI Projects: Measuring the Impact and Demonstrating Value 229
From Model Performance to Business Impact: A PM's Guide to AI Metrics 229
Defining AI/ML-Specific Metrics: The Foundation for Measuring ROI 230
The Importance of Baselines: Knowing Where You Started 230
Understanding the Confusion Matrix: Decoding Classification Performance 231
Key Performance Metrics for AI/ML Models: Beyond the Confusion Matrix 233
Context Matters: Selecting the Right Metrics for Your AI/ML Application 237
1. Define Your Business Goals (and Connect Them to User Needs) 237
2. Consider the Type of AI/ML Application (and Its Inherent Trade-Offs) 238
3. Evaluate the Cost of Errors: The Risk Assessment 239
4. Translate Technical Metrics into Business Impact 240
Important Considerations 240
End-to-End Example-Predicting Churn in a Subscription Service 241
1. Identify the Business Goal: Defining the "Why" 241
2. Define the AI/ML Application and Solution 242
3. Identify Data Sources and Engineer Features: The Raw Materials 243
Available Data 243
Feature Engineering 243
The Product Manager's Role in This Stage 244
4. Select the Metrics: Defining Success 245
The Cost of Errors: Prioritizing What Matters 245
Our Chosen Metrics 246
5. Establish Baseline Metrics: Setting the Starting Point 246
6. Conduct Model Training and Evaluation: Building and Testing the AI 247
7. Conduct A/B Testing: Measuring Real-World Impact 248
8. Calculate the Results and ROI: Quantifying the Value 248
Translating Results into Business Impact 249
Monitoring for Long-Term Success 249
9. Monitor and Maintain the Model for Long-Term Success 250
A/B Testing for AI and ML Projects: Validating Impact and Optimizing Performance 251
What Is A/B Testing (in a Nutshell)? 251
Why Is A/B Testing Especially Important for AI/ML? 252
How to Conduct A/B Testing for AI and ML: A Step-by-Step Guide 253
Key Considerations for AI/ML A/B Testing 258
Chapter Summary and Key Takeaways 259
Key Takeaways 259
Onward: From the Lab to a Live Product 260
Chapter 10 Building and Deploying AI Solutions: From Lab to Live 261
MLOps: The Key to Reliable and Scalable AI 261
Key Components of MLOps-The AI Production Line 264
CI/CD, IaC, and Collaboration: The Foundational Pillars of MLOps 271
Glossary of Key MLOps Terms 272
MLOps End-to-End Example: Churn Prediction in a Subscription Service (Product Manager's Perspective) 274
Chapter Summary and Key Takeaways 278
Key Takeaways 278
Onward: Building with Integrity 279
Chapter 11 Responsible AI and Ethical Considerations: Building AI with Integrity 281
Understanding AI Bias and Fairness: The Foundation of Responsible AI 281
Identifying Potential Biases: Where Bias Can Creep In 282
Mitigating Potential Biases: A Proactive Approach 285
Protected Classes and AI Fairness- Designing for Inclusion 287
What are Protected Classes? 287
Why Focus on Protected Classes? (The Legal and Ethical Imperative) 287
How Protected Classes Relate to AI Bias: The Mechanisms of Discrimination 288
Mitigating Bias Related to Protected Classes: Actionable Steps for PMs 289
AI Ethics and Legal Compliance-From Principles to Practice 291
Understanding the Ethical Landscape: Core Principles 291
Understanding the Legal Landscape: Key Regulations 292
Actionable Steps for Product Managers: Building Ethically and Legally Compliant AI 293
Engaging with the Community and External Stakeholders 297
Chapter Summary and Key Takeaways 298
Key Takeaways 298
Onward: Paving Your Path 299
Chapter 12 Conclusion: Paving Your Own Path to AI PM 301
Embrace Lifelong Learning: Stay Curious and Iterative 302
Cultivate a User-Centric AI Mindset 303
Deepen Cross-Functional Collaboration Skills 303
Build a Distinct AI Portfolio (Show, Don't Just Tell) 304
Develop a Personal Vision for Your AI Career 305
Keep Resilience and Adaptability at the Core 305
Final Thoughts 306
Index 307
CHAPTER 1
Artificial Intelligence and Machine Learning: What Every Product Manager Needs to Know
Stepping into the world of AI product management can feel like learning a new language. You may find yourself in meetings where the conversation suddenly shifts to "training the model," "feature vectors," and "precision versus recall." This technical jargon can be intimidating and can create a barrier between you and your technical counterparts, making it difficult to contribute meaningfully or ask the right strategic questions.
This chapter is designed to break down that barrier. Our goal is to provide you with a solid, PM-focused foundation in the core concepts of Artificial Intelligence and Machine Learning. We will demystify the terminology and explain the fundamental principles in a clear, accessible way, using real-world analogies and product examples.
This isn't about learning to code algorithms; it's about learning to "speak the language" of AI so you can lead your products and teams with confidence. In this chapter, we will clarify the very important difference between AI and ML, explore the major types of Machine Learning, see how models actually learn from data, and walk through the data science lifecycle from a PM's perspective.
AI vs. ML
The terms "Artificial Intelligence" (AI) and "Machine Learning" (ML) are often used interchangeably, but for a Product Manager, understanding their distinction is very important. Think of AI as the broad goal of creating intelligent systems, while ML is a specific and powerful method used to achieve that intelligence.
AI aims to simulate human intelligence in machines to perform tasks like problem-solving, understanding language, and making decisions. Historically, this often involved programming explicit rules. For example, an early chess-playing computer was a form of AI, but it didn't learn; it simply followed a massive set of pre-programmed instructions for every possible move.
Machine Learning (ML), in contrast, is a subset of AI where systems learn directly from data. Instead of being explicitly programmed for every scenario, ML algorithms identify patterns to make predictions and improve their performance over time. This is the key difference. Within ML, Deep Learning (DL) is a specialized branch that employs multi-layered neural networks to automatically learn highly complex patterns from vast amounts of data. Figure 1-1 illustrates the relationship between these concepts.
NOTE Think of teaching a dog a trick. You don't program the dog's muscles; you show examples, give rewards, and the dog learns . ML is similar.
Why This Matters to a PM
Understanding this distinction is not just academic; it directly impacts your day-to-day role as an AI Product Manager in several critical ways:
- Feasibility: Knowing whether a problem is best solved with rule-based AI or data-driven ML helps you assess technical feasibility.
- Resource Allocation: ML requires data-often lots of it. Understanding this helps you plan resources (data collection, labeling, engineering time).
- Iteration Cycles: ML models learn and improve over time. This impacts your product roadmap and release planning.
- User Expectations: ML-powered features can be less predictable than rule-based systems. This requires careful management of user expectations.
Figure 1-1: The relationship between Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL). ML is a subfield of AI, and DL is a specialized subfield within ML.
Key Differences Between AI and ML
To summarize these core distinctions, the following table provides a clear, side-by-side comparison.
FEATURE ARTIFICIAL INTELLIGENCE (AI) MACHINE LEARNING (ML) Scope Broad: Machines simulating human intelligence. Specific: A way to achieve AI by learning from data. Method Can be rule-based, logic-based, or learning-based. Relies on algorithms that learn patterns and make predictions from data. Learning Not always required. Essential; systems improve with experience (data). Example (Product) Smart thermostat with a pre-set schedule. Recommendation engine that learns your viewing preferences and recommends new shows to watch. PM Implication Simpler to implement, less adaptable. More powerful, requires data and iteration.This distinction is key: a PM guiding a simple, rule-based AI feature will have very different planning and resource conversations than one guiding a feature powered by ML. The following examples further illustrate this difference:
- AI (Not ML): A customer service chatbot that uses predefined scripts to answer FAQs. It's "intelligent" in a limited way, but it doesn't learn.
- ML: A spam filter that learns to identify spam emails based on patterns in the text and sender information. It gets better over time as it sees more data.
- AI Powered by ML: Self-driving cars. They use ML extensively (to process sensor data, understand the environment) to achieve the broader AI goal of autonomous navigation.
- AI Powered by ML (Generative AI): Models like DALL-E 2 (image generation) or ChatGPT (text generation). They've learned from vast datasets to create new content.
Common Misconceptions for PMs: Myths vs. Reality
As you begin your AI PM journey, it's important to dispel some common myths. Navigating these misconceptions early will save you time, help you set realistic expectations, and allow you to collaborate more effectively with your technical teams.
MYTH REALITY "AI is a magic black box that just works." AI is applied mathematics and statistics running on data. Its success depends entirely on the quality of the data, the chosen algorithm, and the clarity of the problem definition-all areas where a Product Manager has significant influence. "We need a massive, perfect dataset before we can start." While more high-quality data is better, many impactful AI projects start with smaller, well-curated datasets to build a baseline model (an "AI MVP"). The process is iterative; the model and data can be improved over time. "AI will replace the need for user research." AI amplifies the need for user research. AI can tell you what users are doing, but it can't tell you why. Qualitative user research is essential for understanding the context behind the data and ensuring you're solving the right problem. "Launching an AI feature is a one-and-done project." An AI model is a living system. Its performance can degrade over time as real-world data changes (a concept called "model drift"). AI features require ongoing monitoring, maintenance, and retraining, which must be planned for in your roadmap.Internalizing these realities is a key step in shifting from a traditional software mindset to an AI-first product mindset.
Your Glossary as a PM
To help you speak the language of AI confidently, the following table breaks down the core AI/ML terms you'll need in your toolkit. We'll go beyond simple definitions and provide context to help you truly grasp the concepts:
TERM DEFINITION IN SIMPLE TERMS EXAMPLE Artificial Intelligence (AI) The broad field of computer science focused on creating machines capable of intelligent behavior. Making computers "smart" enough to do things that normally require human intelligence, like problem-solving, learning, and decision-making. A chatbot that understands customer questions and provides helpful answers, or an image recognition system that can identify objects in a photo. Machine Learning (ML) A subset of AI that focuses on enabling machines to learn from data without being explicitly programmed. Teaching computers to learn from examples, so they can make predictions or decisions on their own, without us having to tell them exactly what to do every time. Predicting which customers are likely to churn, or recommending products based on a user's past purchases. Deep Learning (DL) A subfield of ML that uses artificial neural networks with multiple layers (hence "deep") to analyze complex data. A more advanced type of Machine Learning that's particularly good at handling very complex patterns, like those found in images or language. Facial recognition in photos, real-time language translation tools. Algorithm A set of rules or instructions that a computer follows to solve a problem. A recipe that tells the computer what to do step-by-step to achieve a...