Wednesday, May 11, 2011

NPV and ROSI, Part II: Accounting for Uncertainty in the ARO


In a previous post, I proposed a Monte Carlo simulation model that attempts to determine the probability that a security investment will result in a positive Return on Security Investment (ROSI).  The model views security countermeasures and breaches as streams of cash flows and evaluates the Net Present Value (NPV) of each.  To account for the inherent uncertainty in predicting the timing and cost of a breach, the model accepts ranges of possible outcomes and runs repeated simulations.  By calculating the ROSI of many thousands of possible scenarios, the model allows the information security manager to estimate the likelihood that a specific countermeasure will pay for itself by mitigating the impact of a breach.  Please see the earlier post for more detail on this model.

In this follow-up to that post, I propose a slightly different approach to the problem and present a revised version of the simulation tool.  The revised version of the tool includes Annualized Rate of Occurrence (ARO) as one of the variables tested by repeated simulation.

The ARO Challenge

One drawback to my first model is that it requires you to enter a single estimate for the Annualized Rate of Occurrence (ARO). The model then uses this ARO for every subsequent simulation.  If you think that a specific breach happens about once every 3 years, you enter 33% for the ARO, and the tool runs every simulation with a 33% chance of the breach happening each year.

However, determining the ARO of a specific type of breach can be very difficult.  For example, you may be hard pressed to estimate the number of times per year that someone will attempt a war driving attack on your organization’s WLAN.  The ARO will be influenced by a number of factors, including the population density of the area, the accessibility of the campus, and so on.  In spite of your best research, you might still not feel very confident about the ARO that you assign to this event.

Accounting for Uncertainty in the ARO; Restating the Problem

With that challenge in mind, I realized that the ARO itself could be one of the variables in the model.  Instead of asking you for one ARO, I realized I could just have the tool grind out more simulations.  By running the simulation for multiple AROs, you can find a sort of tipping point.  That is, you could find the ARO at which the model begins to produce a positive ROSI in a majority of the simulations.  Put another way, you can determine how frequently the breach has to happen before the security investment makes sense.

This way, the task is not to determine a single ARO but rather to gauge whether the true ARO is probably higher or lower than the tipping point.  Instead of trying to fix a number to some event, you just have to answer the question, “Do I think it will happen more or less frequently than X times per year.”  In many cases, this will be an easier question to answer. 

To use the war driving example from above, you might not be comfortable assigning a specific ARO, but what if any ARO of 33% or higher returns a positive ROSI in a majority of the simulations?  Now your question is “Do I think war driving happens more or less than once every three years?”  This might well be an easier question to answer.

Once you have the tipping point ARO, it may well be unnecessary to spend any more time determining a specific ARO.  In the example above, do you really care much if the ARO is 75% or 90% if an ARO of 33% is sufficient to trigger the decision to buy?

A Revised Simulation Model

The revised model takes this approach (download it from SkyDrive here).  Like the original model, it accepts a range of values for the cost of a breach and the cost of the countermeasure.  However, the revised model now tests AROs from 1% to a maximum set by the user.  The model increments the ARO by 1% for each run of simulations, and records the percent of simulations at each ARO that return a positive ROSI.

After you complete the worksheets for the cost of the breach and the countermeasure, you enter the maximum ARO you want to test.  For example, you might enter a maximum ARO of 300% (you think the event is expected to occur no more than about three times a year).  The model runs thousands of simulations with an ARO of 1%, then repeats the simulations with an ARO of 2%, then 3%, and so on up to 300%.  The results of each run of simulations are displayed.  You might learn than an ARO well below 300% makes the investment a good buy.  On the other hand, you might learn that the investment is only worthwhile if the ARO is implausibly high.

Sample Scenario

I took the example scenario described in my earlier post and fed the same streams of cash flows into the revised model.  I then ran the simulation 2000 times at every ARO between 1% and 200%.  The model ran all the simulations in a minute or two.  The results can be seen in ROSI_Tool_2.xls  (download here).

As you can see, in this example, an ARO of 14% gives about a 50-50 chance that the investment will turn out to be worthwhile.  As the ARO goes higher, the likelihood of the investment paying off naturally increases.  Thus, my question is, “Do I think this event will happen more than about once every seven years (an ARO of 14%)?”  If I answer yes, then I can state that there is a better-than-even chance that the investment will turn out to pay for itself by mitigating a breach at some point in the life of the countermeasure.

Of course, I might want to set the bar higher than a “better-than-even chance” if I’m making a business case for the investment.  In that case, I can simply look to the ARO that produces the appropriate percentage of positive outcomes and ask whether I think the true ARO is above or below that number.


Asking whether the ARO is above or below a certain number is not necessarily an easy question.  It may still be impossible to answer in some cases.  However, it should be easier than attempting to determine and stick to a single ARO. 

I think this approach complements my earlier version of the model.  Once you complete the run of simulations with this version of the tool, you can input specific AROs into the earlier version for more detail on the outcomes, including the graphed results.  The revised model presented here offers a slightly different way of looking at the problem and presents an additional tool the information security manager can use to analyze the business case for a security investment.