Superforecasting Cyber Risk Management

by | Mar 22, 2022

Image Explained: Survey data meshes with ops and claims data to support CISO’s decisions. Surveys are of superforecasters – trained security experts that forecast risk like bookies.

 

“Probability is expectation founded upon partial knowledge.” –– George Boole



Who This Article Is For

This is the third short article in a series on managing cybersecurity risk with a quantitative bent. The first and second articles are optional backgrounds for this post. If you are a risk leader tasked with building or overseeing a cybersecurity risk management program, then this article and its closing calls to action are for you.


 

Introduction: Forecasting Gone Terribly Wrong – The Bay Of Pigs

In 1961 John F Kennedy was contemplating what to do with Cuba. The previous administration had hatched a plan to dislodge the Castro regime using Cuban exiles as invaders. When JFK asked his Chiefs of Staff for an assessment of that plan, they said, “It has a fair chance of success.”

What JFK did not know was that “fair chance” meant a probability of 30%, which to the Chiefs meant “not good.” JFK mistakenly thought “fair chance” meant “good chance” (or high likelihood of success). As they say, the rest is history.1

Note: Kennedy later expressed resentment that his advisors had not given a clear expression of risk prior to the invasion.

Why did JFK’s team avoid giving him a probability? Did probabilities seem overly specific? Too mathy? Maybe the Chiefs felt they lacked hard data thus making probabilities seem unrealistic?

We don’t know if using probabilities would have changed the course of history. What we do know is that the Chief’s use of vague risk terms was wholly misinterpreted by the President. His risk forecasters failed him. A superforecaster would not have made this mistake.

Superforecasters are needed when data is scant (or even non-existent), risks are entangled, and decisions must still be made. Superforecasters never use vague risk terms or scores. Instead, they wield probability to clearly communicate risk.

Unfortunately, vague risk terms and obscure scoring systems are widespread in the enterprise. This pits the superforecaster against entrenched methods that have more in common with astrology than cyber risk management. The good news is that this can be fixed by making the shift from quasi astrological approaches to real risk management.



Shifting From Astrology To Real Risk Management

In antiquity, astrology dominated decision-making. Even today, many cultures still use forms of divination to forecast marriage success, business dealings, and other risky events.

You could say astrology is just an ancient form of risk management – one that’s grounded in pseudoscience (at best).2

This leads to a question: If my risk management approach is more pseudoscience than not, is it equivalent to astrology?

Risk Management Heat Map

Consider this: I place items on a heat map in a mathematically ambiguous way (that prevents ranking and summing risks). If I took that nonsensical approach and began making cyber risk decisions on it, then yes, I may be closer to divination than risk management.

Heat maps, like this one, are pregnant with vague risk terms that lack mathematical grounding. They seem easy to understand – much the same way the phrase “fair chance” appeared obvious, yet contributed to disaster.

Ambiguous approaches typified by heat maps have been proven to make risk management worse.3

The way to correct this is to first treat likelihoods as probabilities – which is partially the focus of this article. Next, treat impacts as dollars. Insurance understands risk this way – exclusively. (I covered this view in my last article.) It’s also the canonical view of risk measurement used in our book How To Measure Anything In Cybersecurity Risk.

 Risk Measurement: A set of possibilities, each with quantified probabilities and quantified losses. For example: “We believe there is a 10% chance that a data breach will result in a legal liability exceeding $10 million.”


This leads to another question: To get “real probabilities” don’t I need years of cyber loss data? The short answer is no. You need superforecasters.


How Do You Make Superforecasters?

Bookies taking bets at horse races in 1941. Warrenton, Virginia

Bookies taking bets at horse races in 1941. Warrenton, Virginia. SOURCE: United States Library of Congress; Photo by Marion Post Walcott

Superforecaster Are Trained: They learn how to use probabilities when assessing risk. They are also trained to use dollars when forecasting impacts. They effectively become security bookies. (see how to get started below)

Superforecasters Are Tested: Their forecasts are graded for stability. If forecasts are quantitatively unstable, they fail.

Superforecasters Workout: Graduate superforecasters are trained ongoingly. They make regular spot forecasts on real risk events that keep their skills sharp.

Superforecasters Compete: The forecasters that have the highest grades over time are used in our risk models. They are tracked on a leaderboard, winning awards and being the envy of their peers for their forecasting prowess.

Superforecasters Multiply: Broad sets of forecasters are used. This includes trained internal forecasters (within Resilience) and beyond. It’s like having an army of security bookies on your team, with more added by the day.

 

What Are We Superforecasting?

1977 tv weatherman with solar energy index

SOURCE: National Archives and Records Administration; Photo by Donald Huebler.

The Value Of Security Controls: We forecast the chance of loss. We find the controls that best reduce those chances. It’s a form of statistical reverse engineering.

The Value Of Insurance: We forecast the chance of overrunning insurance limits. You can adjust limits based on the strength of controls and dollars at risk.

The Optimal Risk Tolerance: As attack surface and dollars at risk expand, insurance and controls are adjusted to keep risk in check. Formally, that is called keeping risk within tolerance. The goal is an optimal tolerance supported by the best return on investment (ROI) for both insurance and controls.


Why Do We Need Superforecasting?

Better (Automated) Decisions: Cyber Risk Management is about making better decisions over time and in time. For the CISO, the decision is what to mitigate, by when, and by how much. The decision is also what to transfer to insurance, by when, and by how much. Intelligent alerts motivate decisions. Alerts fire when risks may exceed tolerance. Superforecaster data informs intelligent alerts.

Better Consensus: When uncertain, we want to know what other people are doing. Consensus gives us assurance and conviction. Unfortunately, most consensus is riddled with bias. Analyst firms with their various “quadrants” and “waves” are influenced by the companies that pay to play. Most CISOs also have confidants (other CISOs) they trust for tough decisions. These confidants are invariably biased. Superforecasting provides a broad-based and quantitatively driven consensus that outperforms biased options.

Better Data: Actuarial tables track similar losses over decades. Conversely, security has dissimilar (and relatively infrequent) losses tracked over short time periods. This means security doesn’t have traditional actuarial tables. The good news is that we do have growing insurance claims data. Superforecasts are meshed with security operations and claims data, outperforming competing data approaches.

 


 

Want To Master Cyber Superforecasting?

Get Trained And Become A Cyber Superforecaster
We are offering Cyber Superforecasting training starting in May. If you are interested in being part of a limited early-bird training (with discount and gratis signed copies of our two books: “How To Measure Anything In Cybersecurity Risk” (July 2016) and The Metrics Manifesto: Confronting Security With Data (March 2022)) simply reach out to get started: cyber-risk-quant@resilienceinsurance.com.


Get Engaged With Cyber Risk Management Consulting
We are offering rapid cyber risk management engagements. The outcome is an optimized cyber risk quantified plan. This is ideal for the security leader looking for a defensible plan now. To get engaged: cyber-risk-quant@resilienceinsurance.com

 

About the Author

Richard Seiersen
Chief Risk Officer

Richard Seiersen is the Chief Risk Officer at Resilience. Prior to joining Resilience in 2021, Seiersen was the co-founder and president of Soluble – a cloud security company sold to Lacework in October 2021. He was previously the Chief Information Security Officer of Twilio, GE Healthcare, and Lending Club. He’s also the co-author of “How To Measure Anything In Cybersecurity Risk” (July 2016) and author of “The Metrics Manifesto: Confronting Security with Data” (March 2022).

 

Sources:

1 Wyden, Peter H. Bay of Pigs: The Untold Story. (1979)

2 Fung, Laura. The Mathematics of Astrology (May 2015)

3  Philip, Thomas, Reidar Bratvold and J. Eric Bickel. The Risk of Using Risk Matrices (Sept. 2013)