CHAPTER 1
Introduction: The Manifesto and the BOOM! Framework

I don't believe in astrology; I'm a Sagittarius and we're skeptical.

- Arthur C. Clarke

We are scientists. We don't blog. We don't twitter. We take our time. bear with us while we think. Comical words found in The Slow Science Manifesto1 written by none other than The Slow Science Academy. Their manifesto serves first and foremost as a reminder to themselves. It defines who they are and aspire to be. It also serves to set public expectations. In this case, "don't expect much, and certainly not via Twitter." And yet they humbly advocate for a point of view, "Science needs time to think. Science needs time to read, and time to fail. Science does not always know what it might be at right now." A good manifesto does all these things. It creates identity for its signees, it sets expectations for its audience, and it advocates for a point of view without being too much of a bully.

The Metrics Manifesto strives to do all those things, particularly the last one - a point of view without too much tyranny. It endeavors to be a framework for creating simple security metrics and advanced ones for those who need them. Lastly, this book serves as a guide for making a complete enterprise security metrics program - a program that is grounded in the principle of confronting security with data.

One caveat before you start: A manifesto typically outlines a minority position. You wouldn't need a manifesto if everyone already agreed with you. And while this is a minority position relative to security, it's likely a majority position with measurement professionals at large. The term measurement professionals includes scientists, actuaries, statisticians, engineers, and others. This is the group of people who seemed to align with our previous book titled How to Measure Anything in Cybersecurity Risk (Wiley 2016). The actuaries really liked it! In 2018, it was required reading for The Society of Actuaries exam prep. While the Manifesto is quite different from that work, it fully aligns in measurement spirit. Be forewarned: The methods herein may be foreign and at times challenging. When you feel unsure, just imagine measurement experts past and present cheering you on!

Lastly, I do hope "The Manifesto" produces productive skepticism about security. It should come naturally to us. After all, security professionals poke and prod to discover why someone else's digital ideas are risky. Shouldn't a true skeptic turn that same confrontational mindset on themselves and muse, "This security capability I've deployed may not work. What would I see occurring that would let me know if it does??" That's the first step in confronting security. It's the first step in designing a powerful security metrics system that makes a significant difference in our battle against our adversaries.

What's Next: Caveats and Epiphanies

The next section covers the manifesto and the BOOM Framework. The manifesto is built around four key observations. Each observation, in turn, has one or more supporting beliefs. You don't need to become a convert to those beliefs. In fact, you should maintain doubt. That would line up with the theme of "confronting security with data."

The BOOM Framework is built around five key baselines. And these five baselines each get hefty chapters dedicated to them. Now for some caveats.

The first caveat is that skipping chapters will be rough without the right background. That's why I recommend reading the whole book.

My next caveat is the same one Doug Hubbard and I made in our last book: This is not a statistics book. What is it then? It's a metrics book. I know it seems obvious to say that - but I think certain readers appreciate being forewarned. If you are coming here looking for the latest, greatest, in-depth quantitative stuff, then this may not be the book for you. That being said, there may be some perspectives that even seasoned data scientists, statisticians, and others may find of interest.

Next to the last caveat: This book has code. The good news is that much of the code is in the form of one-liners (some lines might be quite long due to clever tricks of the trade). It's not my plan to turn you into a data scientist. First, I don't think I am qualified. Second, it's completely unnecessary. Why become a carpenter when you only need to use a hammer?

Last caveat: This one is on my qualifications to write a book with such a lofty title. My qualifications are that I am well acquainted with operational sadness. I've spent most of my career in the foxhole - both on the vendor side and in operations at varying organizational levels. The whole time, I couldn't shake the feeling that there must be a better way to manage operations - particularly security. Dissatisfied, and prone to wander, I started to look outside of security and even technology. My question was, "Who else was solving big problems where uncertainty abounds and the risks are real??" This is when I started running across people I will refer to as measurement experts.

Measurement experts are the humble statisticians, natural scientists, decision analysts, and other folks tackling seemingly impossible-to-measure problems. They are all decidedly more educated than me, and a few can sling code really well. But what I bring to the kitchen table, and you do too, is operational experience within a problem domain. Once I relaxed my prejudices about my lack of quantitative savvy, I started to become productive - dare I say creative. This newfound freedom led to the following "epiphanies of the obvious."

Epiphanies of the Obvious

• Computers are very good at math. So good at it, in fact, that you don't need to be. Understanding calculus has great utility, but your computers often know enough for most of your problems.
• Measurement experts create their wares for other people to use. This is similar to the cobbler who makes shoes for others to wear - just not for himself. There is a vast array of freely available analytic tools made for people like you and me.
• Problem understanding is job #1, not quantitative skills. My favorite quote by Charles Kettering reads, "A problem well defined is a problem half solved." Once you have framed your problem well, you can usually pull down the code you need from the spice rack of analytics to start baking. If you happen to have a problem that is out of reach, call an expert - 99% of the problems you have are likely in reach if you spend enough time framing your problem and being resourceful.
• The last epiphany (which we covered in the first book) was that our goal should be to "beat the competing model." I just need to be slightly better than the competition, not perfect. In short, when you think you must have perfect math, code, statistics, i.e. "metrics" then you are bound to the fate of Sisyphus. He was the king cursed to push a boulder up a hill for all eternity. I think this mindset may be the blocker that prevents security teams from taking things to the next level in operational excellence, I am very sad to say.

Thus, my ambitions for this book, and for you, remain humble. I merely want to beat the competing model for security metrics. That model is typically just a list of basic counts of things and not much more. Don't get me wrong - lists of metrics and counts are not necessarily bad. In fact, they are necessary. I just know we can do better together.

Next up is "The Manifesto" and the BOOM! Metrics Framework.

The Metrics Manifesto and BOOM!

A lot of people have problems with public confrontation, but it doesn't worry me at all. I can handle myself. I know my martial arts. - Pink

This section presents the pithy "Metrics Manifesto." It's less than a page in length. Think of it as the ethos, or spirit, behind the book. I encourage re-reading it from time to time.

The rest of the chapter outlines the BOOM! Framework. It's the metrics framework that evolved out of the aforementioned speed consulting. It also provides an outline for the book. With BOOM, you will encounter interesting measurement methods like survival analysis, burndown rates, arrival analysis, interarrival analysis, escape rate, Bayesian data analysis, and more. Each method is designed to help you confront your security program with data. Taken as a whole, these methods embody the Metrics Manifesto.

The (Modern) Metrics Manifesto

Observation: Most metrics count; the best ones confront.

• Belief: "We believe shrinking attack surface, while not slowing value exposure, is the new job #1 for security."
• Belief: "We also believe not doing this gives advantage to our adversaries and reduces business opportunity."

Observation: Most metrics reveal what is certain; the best ones also retain what is uncertain.

• Belief: "We believe metrics that ignore our uncertainty ignore our adversaries."