Chapter 1

Catching the Data-Mining Train

You’ve picked an exciting moment to become a data miner.

By some estimates, more than 15 exabytes of new data are now produced each year. How much is that? It’s really, ridiculously big — that’s how much! Why is this important? Most organizations have access to only a teeny, tiny fraction of that data, and they aren’t getting much value from what they have.

Data can be a valuable resource for business, government, and nonprofit organizations, but quantity isn’t what’s important about it. A greater quantity of data does not guarantee better understanding or competitive advantage. In fact, used well, a little bit of relevant data provides more value than any poorly used gargantuan database. As a data miner, it’s your mission to make the most of the data you have.

This chapter goes over the basics of data mining. Here I explain what data miners do and the tools and methods they use to do it.

Getting Real about Data Mining

Maybe you’ve heard news reports or ads hinting that all you need to make valuable information pop out like magic is a big database and the latest software. That’s nonsense. Data miners have to work and think to make valuable discoveries.

Maybe you’ve heard that to get results out of your database, you must first hire one of a special breed of people who have nearly super-human knowledge of data, people known to be very expensive, nearly impossible to find, and absolutely necessary to your success. That’s nonsense, too. Data miners are ordinary, motivated people who complement their business knowledge with the fundamentals of data analysis.

Data mining is not magic and not art. It’s a craft, one that mere mortals learn every day. You can find out about it, too.

Not your professor’s statistics

Perhaps you took a class in statistics a long time ago and felt overwhelmed by the professor’s insistence on rigorous methods. Relax. You’re out to find information to support everyday business decisions, and many everyday business problems can be solved using less formal analysis methods than the ones you learned at school. Give yourself some slack.

How do you give yourself slack? By data mining, that’s how.

Data mining is the way that ordinary businesspeople use a range of data analysis techniques to uncover useful information from data and put that information into practical use. Data miners use tools designed to help the work go quickly. They don’t fuss over theory and assumptions. They validate their discoveries by testing. And they understand that things change, so when the discovery that worked like a charm yesterday doesn’t hold up today, they adapt.

The value of data mining

Business managers already have desks piled high with reports. Some have access to computer dashboards that let them see their data in myriad segments and summaries. Can data mining really add value? It can.

Typical business reports provide summaries of what has happened in the past. They don’t offer much, if anything, to help you understand why those things happened, or how you might influence what will happen next.

Data mining is different.

Here are examples of information that has been uncovered through data mining:

• A retailer discovered that loyalty program sign-ups could be used to identify which customers were most likely to spend a lot and which would spend a little over time, based on just the information gathered on the customer’s first visit. This information enabled the retailer to focus marketing investment on the high spenders to maximize revenue and reduce marketing costs.
• A manufacturer discovered a sequence of events that preceded accidental releases of toxic materials. This information enabled the manufacturer to keep the facility operating while preventing dangerous accidents (protecting people and the environment) and avoiding fines and other costs.
• An insurance company discovered that one of its offices was able to process certain common claim types more quickly than others of comparable size. This information enabled the insurance company to identify the right place to look for best practices that could be adopted across the organization to reduce costs and improve customer service.

Data mining helps you understand how the elements of your business relate to one another. It provides clues about actions that you can take to make your business run more smoothly and generate more revenue. It can help you identify where you can cut costs without damaging the organization, and where spending brings the best returns.

Data mining provides value by helping you to better understand how your business works.

Trust data or trust your gut?

Can intuition tell you what motivates people to buy, donate, or take action? Many people believe that no data analysis can outdo their own gut feel for guiding decisions.

I challenged business managers to put their intuition to the test. They came from a variety of industries, businesses small and large, and included both young and experienced managers. Each viewed ten pairs of ads like these:

• Two nearly identical ads, differing only in that one showed a female face and the other a male. Which generated more leads?
• An ad with many images was contrasted with one that had just a few. Which one resulted in more purchases?
• Two ads had the same copy (text) but different layouts. Which would draw more donations for a charity?

Small variations in images, layout, or copy can make dramatic differences in an ad’s effectiveness. Tests of the samples in this guessing game demonstrated that the right choice could lift conversions (actions on the part of the customer, such as buying, donating, or requesting information) by 10 percent, 30 percent, and sometimes more. In one case, the superior ad resulted in 100 percent more conversion than the alternative.

Could anyone tell, just by looking, which alternatives would perform best? No. None of the managers were effective at picking the best ads. Flipping a coin worked just as well.

If you want to make good business decisions, you need data. Use your brain, not your gut!

Working for it

A lot of people have unrealistic expectations about data mining. That’s understandable, because most people get their information about data mining from people who have never done it.

Some people expect data mining to be so easy that they will only need to feed data into the right software and a tidy summary of valuable information will automatically pop out. On the other hand, some expect data mining to be so difficult that only someone with expert programming skills and a Ph.D. in physics can tackle it. Some expect data mining to produce great results even if the data miner doesn’t know what anything in the data means. These are all unrealistic expectations, but they’re understandable. News reports, sales pitches, and misinformed people often circulate ideas about data mining that are just plain wrong. How is anyone to know what’s reasonable and what’s hype?

Here’s what’s realistic: Many novice data miners find that a few days of training and a month of practicing what they have learned (part-time, while still performing everyday duties) are enough to get them ready to begin producing usable, valuable results. You don’t need to have a mind like Einstein’s, a Ph.D., or even programming skills. You do need to have some basic computer skills and a feel for numbers. You must also have patience and the ability to work in a methodical way.

Data mining is hard work. It’s not hard like mining coal or performing brain surgery, but it’s hard. It takes patience, organization, and effort.

Doing What Data Miners Do

If you think of data as raw material, and the information you can get from data as something valuable and relatively refined, the process of extracting information can be compared to extracting metal from ore or gems from dirt. That’s how the term data mining originated.

Do the words data miner conjure up a mental image of a gritty worker in coveralls? That’s not so far off the mark. Of course, nothing is physically dirty about data mining, but data miners do get down and dirty with data. And data mining is all about power to the people, giving data analysis power to ordinary businesspeople.

Focusing on the business

Data miners don’t just ponder data aimlessly, hoping to find something interesting. Every data-mining project begins with a specific business problem and a goal to match.

As a data miner, you probably won’t have the authority to make final business decisions, so it’s important that you align your work with the needs of decision makers. You must understand their problems, needs, and preferences, and focus your efforts on providing information that supports good business decisions.

Your own business knowledge is very important. Executives are not going to sit next to you while you work, providing feedback on the relevance of your discoveries to their concerns. You must use your own experience and acumen to judge that for yourself as you work. You may even be familiar with aspects of the business that the executive is not, and be able to offer fresh perspectives on the business problem and possible causes and...