Metrics

An incomplete set of thoughts triggered by Gunnar's blog...

Gunnar Peterson blogged about Dan Geer's synopsis of MetriCon. On some points, I disagree with Gunnar and Dan both.

We do have quite a few security metrics. It's just that they're often disguised as other things: router traffic levels, service load graphs, inventories, trouble ticket systems, personnel management systems, etc.

To take of Gunnar's point, "metric" is also a particularly harmful word in that not everyone understands what a metric is. Yes, it is something that is measured over time. (This is where most people's definition stops.) It also includes the processes for interpreting the gathered data. This can be in the form of: overlaying a daily average or an acceptable range, setting trigger points based on sustained levels, setting priorities based on levels of non-compliance, etc.

In other words, it's not just the collection of data. It's also how you use that data (e.g., what decisions you base on the data or what do you calculations control). You also have to decide how you're going to federate that data (single-purpose metrics are rare).

People get into serious trouble making assumptions about security metrics (i.e., "we need them!") without defining "what the job is".

To better design your metrics:

• First, determine what decisions need to be made. If they're management level decisions, they should be very broad and generic. If they're technical level decisions, they should be very specific and rigin.
• Next decide what set of questions relates to each of those decisions. Each question should be simple. Examples include: "how many" or "how often").
• Then determine what temporal data sets are available to answer those questions. (Keep in mind that whatever the data is, it needs to be tracked over time.) This step is often the most difficult as the available data is often outside of the local knowledge base (e.g., in someone else's department or organizations) even though it is often readily available.
• Lastly (and most importantly), train people (or hire 'em) to use those metrics. The majority of metrics already available rot on the vine, ignored by the people who most need them. Your high-level metrics will affect the most people and will likely require the most training and tend to support long-term decisions. Your low-level metrics will be the most technical, will be used by the fewest people, will be the least visible, and will tend to support repetitive high-speed (daily, hourly, etc.) decision.