Analytics the Right Way
Description
CLEAR AND CONCISE TECHNIQUES FOR USING ANALYTICS TO DELIVER BUSINESS IMPACT AT ANY ORGANIZATION
Organizations have more data at their fingertips than ever, and their ability to put that data to productive use should be a key source of sustainable competitive advantage. Yet, business leaders looking to tap into a steady and manageable stream of "actionable insights" often, instead, get blasted with a deluge of dashboards, chart-filled slide decks, and opaque machine learning jargon that leaves them asking, "So what?"
Analytics the Right Way is a guide for these leaders. It provides a clear and practical approach to putting analytics to productive use with a three-part framework that brings together the realities of the modern business environment with the deep truths underpinning statistics, computer science, machine learning, and artificial intelligence. The result: a pragmatic and actionable guide for delivering clarity, order, and business impact to an organization's use of data and analytics.
The book uses a combination of real-world examples from the authors' direct experiences-working inside organizations, as external consultants, and as educators-mixed with vivid hypotheticals and illustrations-little green aliens, petty criminals with an affinity for ice cream, skydiving without parachutes, and more-to empower the reader to put foundational analytical and statistical concepts to effective use in a business context.
More details
Other editions
Additional editions
Persons
TIM WILSON has been an analytics practitioner since 2001, working in roles from business intelligence at high-tech B2B companies, to analytics leadership at marketing agencies, to consulting with Fortune Global 500 companies to improve their analytics investments.
DR. JOE SUTHERLAND has worked as an executive, public servant, and educator for the Dow Jones 30, The White House, and our nation's top universities. His firm, J.L. Sutherland & Associates, has attracted clients such as Box, Cisco, Canva, The Conference Board, and Fulcrum Equity Partners. He founded the Center for AI Learning at Emory University, which focuses on AI literacy and integration for the general public.
Content
Acknowledgments xiii
About the Authors xvii
Chapter 1 Is This Book Right for You? 1
The Digital Age = The Data Age 3
What You Will Learn in This Book 6
Will This Book Deliver Value? 7
Chapter 2 How We Got Here 9
Misconceptions About Data Hurt Our Ability to Draw Insights 11
Misconception 1: With Enough Data, Uncertainty Can Be Eliminated 12
Having More Data Doesn't Mean You Have the Right Data 13
Even with an Immense Amount of Data, You Cannot Eliminate Uncertainty 16
Data Can Cost More Than the Benefit You Get from It 18
It Is Impossible to Collect and Use "All" of the Data 18
Misconception 2: Data Must Be Comprehensive to Be Useful 19
"Small Data" Can Be Just As Effective As, If Not More Effective Than, "Big Data" 20
Misconception 3: Data Are Inherently Objective and Unbiased 21
In Private, Data Always Bend to the User's Will 23
Even When You Don't Want the Data to Be Biased, They Are 24
Misconception 4: Democratizing Access to Data Makes an Organization Data-Driven 26
Conclusion 28
Chapter 3 Making Decisions with Data: Causality and Uncertainty 29
Life and Business in a Nutshell: Making Decisions Under Uncertainty 30
What's in a Good Decision? 32
Minimizing Regret in Decisions 33
The Potential Outcomes Framework 34
What's a Counterfactual? 34
Uncertainty and Causality 36
Potential Outcomes in Summary 42
So, What Now? 43
Chapter 4 A Structured Approach to Using Data 45
Chapter 5 Making Decisions Through Performance Measurement 53
A Simple Idea That Trips Up Organizations 54
"What Are Your KPIs?" Is a Terrible Question 58
Two Magic Questions 60
A KPI Without a Target Is Just a Metric 68
Setting Targets with the Backs of Some Napkins 72
Setting Targets by Bracketing the Possibilities 74
Setting Targets by Just Picking a Number 78
Dashboards as a Performance Measurement Tool 80
Summary 82
Chapter 6 Making Decisions Through Hypothesis Validation 85
Without Hypotheses, We See a Drought of Actionable Insights 88
Breaking the Lamentable Cycle and Creating Actionable Insight 89
Articulating and Validating Hypotheses: A Framework 91
Articulating Hypotheses That Can Be Validated 92
The Idea: We believe [some idea] 95
The Theory: ...because [some evidence or rationale]... 96
The Action: If we are right, we will... 98
Exercise: Formulate a Hypothesis 101
Capturing Hypotheses in a Hypothesis Library 101
Just Write It Down: Ideating a Hypothesis vs. Inventorying a Hypothesis 104
An Abundance of Hypotheses 105
Hypothesis Prioritization 106
Alignment to Business Goals 107
The Ongoing Process of Hypothesis Validation 108
Tracking Hypotheses Through Their Life Cycle 109
Summary 110
Chapter 7 Hypothesis Validation with New Evidence 113
Hypotheses Already Have Validating Information in Them 115
100% Certainty Is Never Achievable 116
Methodologies for Validating Hypotheses 118
Anecdotal Evidence 119
Strengths of Anecdotal Evidence 120
Weaknesses of Anecdotal Evidence 121
Descriptive Evidence 122
Strengths of Descriptive Evidence 123
Weaknesses of Descriptive Evidence 124
Scientific Evidence 128
Strengths of Scientific Evidence 129
Weaknesses of Scientific Evidence 135
Matching the Method to the Costs and Importance of the Hypothesis 137
Summary 139
Chapter 8 Descriptive Evidence: Pitfalls and Solutions 141
Historical Data Analysis Gone Wrong 142
Descriptive Analyses Done Right 146
Unit of Analysis 146
Independent and Dependent Variables 149
Omitted Variables Bias 151
Time Is Uniquely Complicating 153
Describing Data vs. Making Inferences 154
Quantifying Uncertainty 156
Summary 163
Chapter 9 Pitfalls and Solutions for Scientific Evidence 165
Making Statistical Inferences 166
Detecting and Solving Problems with Selection Bias 168
Define the Population 168
Compare the Population to the Sample 168
Determine What Differences Are Unexpectedly Different 169
Random and Nonrandom Selection Bias 169
The Scientist's Mind: It's the Thought That Counts! 170
Making Causal Inferences 171
Detecting and Solving Problems with Confounding Bias 172
Create a List of Things That Could Affect the Concept We're Analyzing 173
Draw Causal Arrows 173
Look for Confounding "Triangles" Between the Circles and the Box 174
Solving for Confounding in the Past and the Future 175
Controlled Experimentation 176
The Gold Standard of Causation: Controlled Experimentation 177
The Fundamental Requirements for a Controlled Experiment 179
Some Cautionary Notes About Controlled Experimentation 184
Summary 185
Chapter 10 Operational Enablement Using Data 187
The Balancing Act: Value and Efficiency 189
The Factory: How to Think About Data for Operational Enablement 191
Trade Secrets: The Original Business Logic 192
How Hypothesis Validation Develops Trade Secrets and Business Logic 193
Operational Enablement and Data in Defined Processes 194
Output Complexity and Automation Costs 196
Machine Learning and AI 199
Machine Learning: Discovering Mechanisms Without Manual Intervention 199
Simple Machine-learned Rulesets 200
Complex Machine-learned Rulesets 202
AI: Executing Mechanisms Autonomously 203
Judgment: Deciding to Act on a Prediction 204
Degrees of Delegation: In-the-loop, On-the-loop, and Out-of-the-loop 204
Why Machine Learning Is Important for Operational Enablement 209
Chapter 11 Bringing It All Together 211
The Interconnected Nature of the Framework 212
Performance Measurement Triggering Hypothesis Validation 212
Level 1: Manager Knowledge 213
Level 2: Peer Knowledge 214
Level 3: Not Readily Apparent 215
Hypothesis Validation Triggering Performance Measurement 216
Did the Corrective Action Work? 216
"Performance Measurement" as a Validation Technique 216
Operational Enablement Resulting from Hypothesis Validation 220
Operational Enablement Needs Performance Measurement 222
A Call Center Example 223
Enabling Good Ideas to Thrive: Effective Communication 225
Alright, Alright: You Do Need Technology 226
What Technology Does Well 227
What Technology Doesn't Do Well 228
Final Thoughts on Decision-making 230
Index 233
Chapter 1
Is This Book Right for You?
You picked up this book, which means you're thinking that something about the way you and your organization use data and analytics is not "right." Time and again, the executives, managers, and new hires who make up our clients, colleagues, and friends have expressed to us their anxieties related to how they and their teams are using data and analytics:
"We have plenty of data, but the actionable insights we get from it are few and far between."
"Our team consistently invests in the latest data tools and platforms to ensure we're collecting and storing all the data we might need, but the recommendations we generate from those data never really increase in quality or volume."
"We work with agencies and consultancies that do a lot of reporting on the results they're delivering for us. Those tend to be lengthy presentations with a ton of charts, but I often feel like I'm just having data thrown at me that may or may not be representing real business value being delivered."
"I never feel comfortable investing the millions we invest in paid media; it's unclear if we're actually getting the returns our agencies report, or if they just tortured the data until it confessed a positive answer."
"We have talented analytics and data science teams, but it feels like we're talking past each other when I interact with them. I really need them to generate insights and recommendations, and they seem frustrated when I tell them that that's not what they're providing."
"My data engineers over-promise what their machine learning and AI techniques can do for our stakeholders; it tanks our credibility when we promise magic but don't understand the nuts and bolts well enough to do it right."
"My product teams build these exotic proofs-of-concept using the latest and greatest AI tools. But to scale them up is way too expensive, and the production engineers tasked with doing so can't understand the opaque mathematical techniques being used."
"Our technology platform partners sell us licenses to their latest technology and their latest AI or machine learning, and they share eye-popping stories for how effective they are. But when we dig into the pilots, the platforms don't offer anything more than what we're already doing. I wish I could see through these sales pitches earlier."
"We have a ton of automated dashboards, and I understand most of the data that they include, but I still struggle to figure out how I should be using that data to make decisions. Where do I start?"
If any of these quotes feel familiar, then this book is for you. We've heard these frustrations in every data-related function in nearly every industry, ranging from pharmaceuticals to health care, retail, financial services, and consumer packaged goods. And we've worked with clients in all of these industries to shift their approaches. Putting your data to use can be productive, profitable, and even fun! That's why we wrote this book: to guide business leaders who want to use their data effectively.
The Digital Age = The Data Age
A common theme across all of the frustrations we hear from organizations about their struggles to effectively and consistently extract meaningful business value from their investments in data and analytics is that, well, there's just so much data. Our instincts have long been that more data is better, but the shifting of all aspects of our lives from analog to digital over the past three decades has wrought such an extreme version of "more" that it has left many managers questioning those instincts. The origins of the internet are often traced back to the mid-1960s and the creation of ARPANET as a distributed control computer network funded by the US Department of Defense. It was not until 1989, though, that Tim Berners-Lee at CERN conceived of an easier-to-use evolution of what had become "the internet" that would become the "World Wide Web." Within four years, Marc Andreesen, a student at the University of Illinois Urbana-Champaign created the Mosaic web browser while working with the National Center for Supercomputing Applications (NCSA), and the internet was on its way to catching mainstream fire. From the several hundred websites that existed by the end of 1993, to the more than 20,000 in 1995, to 17 million in 2000,1 the growth of digital content was exponential.
Organizations began transitioning every aspect of their businesses to digital formats. Digital bits and bytes trumped paper on countless fronts: storability (a room full of file cabinets was replaced with a thumb drive), searchability (leafing through those file cabinets pulling out folder after folder and scanning the pages within those folders was replaced by a rectangle on a computer screen into which keywords could be typed), portability (traipsing to the library or the records room or a coworker's office was replaced by launching a browser from any device connected to the internet, and seemingly every device is connected to the internet). At a macro scale, global life began going through an analog-to-digital conversion:
- Rather than sending a letter, we could send an email.
- Rather than going to a brick-and-mortar establishment to buy a book, or leafing through a publisher's quarterly catalog, we could search for one online and order it immediately.
- Rather than receiving a book in the mail, we could read it instantaneously in a digital format.
- Rather than advertising on billboards, in magazines and newspapers, or with direct mail, we could advertise on the personalized screens that consumers were spending more and more time looking at, by running ads on websites and search engines.
- Rather than staffing a customer service representative to help prospects find what they need, we could use data science to offer our customers personalized recommendations in real time.
As early as 1994, BusinessWeek reported, "Companies are collecting mountains of information about you, crunching it to predict how likely you are to buy a product, and using that knowledge to craft a marketing message precisely calibrated to get you to do so [.] Many companies were too overwhelmed by the sheer quantity of data to do anything useful with the information [.] Still, many companies believe they have no choice but to brave the database-marketing frontier."2 The digital data revolution was in full swing.
For companies, perhaps the most exciting aspect of this pervasive transformation to a digital-first world was the increased scale and fidelity of the data that could be collected along the way. Ask a retailer how their customers walk through one of their physical stores, and they would have to hire a set of observers to position themselves in the store and take copious notes. And they would only have data for the periods when those observers were on site. And they would run the risk of affecting their customers' behavior in the process, the so-called "observer effect." Ask a retailer how their customers navigate their website, though, and they are just a few clicks away from being able to pull up a report in a digital analytics platform like Google Analytics.
Expectations were high. With all of this data, it seemed obvious that amazing things were possible! And amazing things can be done with data. But over the last 25 years, businesses have slid into what Matt Gershoff, the chief executive officer of Conductrics, refers to as a "big table mentality." They have begun the never-ending and ever-increasing pursuit of gathering "all" the data-striving to clean, store, integrate, and maintain all of the data has become a goal in and of itself. "We can predict, discover, and engineer anything, if only we can observe everything," the philosophy suggests. "We're going to be truly scientific with all of this data" is the idea, but a misunderstanding of scientific principles and their application leads to ineffective and frustrating results rather than the "actionable truths" that we expected.
We (the authors) absolutely believe in the value of data. But we also have personally observed the negative results of flawed approaches and misguided expectations of how to realize that value. These negative results force leaders to seek external correction (many times, that's where we would be hired), or in the worst cases, lose the trust of their customers. We hope more leaders will proactively seek the knowledge this book offers to start reversing this trend. If you've ever felt frustrated, fascinated, forestalled, or fired up with the industry of data and analytics, this book is for you.
What You Will Learn in This Book
We are data enthusiasts, and we believe that data and analytics have near limitless extraordinary potential. But we have seen that the best intentions to put data to productive use can still lead to ineffective and even destructive activities. In this book, we will give you the tools to use data to enable effective decision-making and automation with clarity and purpose.
In Chapter 2, we explore the root causes that have led many organizations to invest extraordinary amounts in their data infrastructure, in reporting and analysis...
