Chapter 1

The Big Picture of Big Data Jobs

In This Chapter

Understanding why big data is important today

Discovering the available career paths

Finding out what kinds of firms hire big data professionals

Some people have said that information is the new oil. There is a wealth of value locked up inside this new black gold. As with oil, the challenge is finding it, extracting it, and converting it to something useful. Information empowers new markets, innovations, and even transformation of societies. Like oil exploration, the challenge is discovering how to unlock potential value deep inside an ocean of data. That's the art and science of big data.

Big data has gone beyond the buzzword phase and into driving real value for organizations around the world. The Boston Consulting Group recently conducted a groundbreaking study that found a correlation between the use of big data and bottom-line revenue. It studied 167 companies in five sectors - financial services, technology, consumer goods, industrial goods, and other services - and found that those that worked with big data increased overall revenue for their firms by as much as 12 percent. Those are real dollars! The study concluded that leaders in innovation are more likely to credit big data as a significant contributor to their growth.

That's precisely why the market is seeing a significant uptick in demand for big data professionals. Firms are scrambling to hire knowledge workers who can help find new information wells of value locked up inside these vast fields of data. In this chapter, I explain why big data has arrived on the scene and what that means for career paths in this exciting new discipline.

How We Got Here and Where We're Headed

Why is big data such a big deal? You may be asking, "Didn't we always have lots of data with huge databases?" You may even be working on a DB2 mainframe database with data going back to the 1970s! Does that mean you're using big data? You may or may not be. When your datasets become so large that you have to start innovating around how to collect, store, organize, analyze, and share it, you're using big data.

Big data has come into the spotlight because of the convergence of two significant developments in recent years:

• There has been a substantial increase in variety, volume, velocity, and veracity of data. We call that the four V's of big data. I add a fifth - value.
  • Volume: How big the datasets are. Defining volume in terms of terabytes wouldn't be very helpful because datasets are growing every year. Consider high-definition video as an example: Each second of video requires 2,000 times more bytes than a single page of text. A 20-minute ultra-high-definition uncompressed video requires roughly 4 terabytes (TB) of storage. You get the picture.
  • Variety: The different types of data formats included in your dataset. This is the attribute that comes to mind when people think about big data. Traditional data types (called structured data), including things like date, amount, and time, fit neatly in a relational database (a database where the information is arranged in columns so that they can be compared). But big data also includes unstructured data (data that doesn't have a predefined model or isn't organized in a predictable manner). It includes things like Twitter feeds, audio files, MRI images, web pages, and anything that can be captured and stored but doesn't have a meta model (a model that describes what the data is made up of) that neatly defines it.
  • Velocity: The high rate at which data flows into an organization or system. Think of streaming video data from a security camera or tick data from a financial exchange. Velocity isn't a new idea. What makes it special in big data is the capability to sift through the information very quickly in near-real time. The trick is sifting the noise.
  • Veracity: One of the key concerns of all managers is whether the data is accurate. Can they use it to make predictions? Inherent in all data are inaccuracies. Does this data have more inaccuracies than expected?

  In addition to these four elements, I like to add a fifth V, value, which is the convergence of these four elements. Technology without value is just cool. What makes big data such an innovation is the fact that the intersection of these four V's generates tremendous value. It may not make the typical diagrams, but I certainly think it should.

• The technical capability now exists to capture, store, and process this data into meaningful information quickly. New data is being generated at a much higher rate today than in the past. For example, according to MIT Technology Review, in 2012 there were 2.8 zettabytes (ZB) of data but that number was projected to double by 2015. The advent of cloud technology, low-cost massive computing engines, and new innovations in data capture and analysis tools have made the capture and storage of this data a technically achievable goal.

Some examples of these datasets include

• IT, application server logs: IT infrastructure logs, metering, audit logs, change logs
• Websites, mobile apps, ads: Clickstream, user engagement
• Sensor data and machine-generated data: Weather, smart grids, wearables, cars
• Social media, user content: Messages, updates

As this field progresses, the amount of data, sensor points, and information will continue to trend up, as will our ability to mine this data for valuable and actionable information - information that gives managers the ability to make decisions about a business, product, or industry. What this means for you is that the job market will continue to see an increase in both demand and function for big data professionals.

Why companies care about big data

Companies care about big data because the promise of big data is transformational. The potential savings, new revenues, and innovations are limitless. For example, McKinsey & Company predicts that in healthcare alone, the application of big data has a potential value of $300 billion to the U.S. healthcare system, which is two times the annual healthcare spending in Spain. Organizations have realized that big data will increase their capability to compete by lowering costs or uncovering new revenue streams. Simply put, big data impacts the bottom line in a big way.

McKinsey & Company is a global management consulting firm with more than $7 billion in revenue and more than 13,000 employees. It serves as a key advisor to the world's leading companies and governments. Some of its influential publications include McKinsey Quarterly and research from the McKinsey Global Institute. Its 2010 research on big data became one of the major levers in driving global awareness to the potential of this new field.

The future of big data jobs

As an industry explodes, so do the job opportunities. The required functions of big data range from back-end systems administrators and model designers to front-end business analysis. The jobs can be for anyone from folks who are less technically inclined but have strong marketing skills to hard-core math wonks and everything in between. There is good evidence to suggest that many of the jobs will be located within the borders of one's own country. It is difficult to outsource big data jobs. One of the reasons for this is the fact that it is both difficult and expensive to move massive amounts of people around the globe. The requirement to be co-located near a business unit or field team is critical (see Chapter 4). A quick search on popular online job sites shows thousands of available big data jobs in the United States.

Exploring Big Data Career Paths

The types of roles in big data are many, but they do share some common attributes. And don't worry: They don't all require a PhD in math or statistics.

Not everyone is a data scientist

So, what is a data scientist? She is practitioner who helps the company achieve a competitive advantage through the use of the data. When the big data field began to emerge, people quickly jumped at labeling what they thought the corresponding job function would be. The term data scientist was thrown around in IT circles, but people weren't really sure what that job would look like. What emerged was the idea that big data can only be done by the most advanced mathematicians, statistical modelers, and specialized programmers. For many people, images of a Wall Street quantitative analyst comes to mind. (A quantitative analyst, or quant, is someone who uses models to determine when to buy and sell specific stocks.)

There continues to be a demand for traditional data scientists, but the field has expanded to include a broad spectrum of functions - in part because the advancement of technology has made using big data systems easier (see Chapter 7 for more on big data tools).

Thoughts from an experienced business analyst

I had an early interest in computing and technology when I was younger, but I really got started with data and analytics while pursuing an M.S. in management information systems at the...