I have found that most of my customers are trying to assign metrics to things that are still dynamic and moving. I see metrics thrown all over the place that end up being useless and adds more to the confusion when trying to understand the true health of a company’s security position. In my opinion, the industry is still struggling with how to properly use metrics because we have not properly defined how to develop a security program or security enterprise architecture or the items that make them both up.

When I layout a strategic security program and enterprise architecture for a company to get to - assigning metrics is one of the last components.

You can’t measure a process that is chaotic or a process that is not matured and repeatable. Applying metrics to purely technical items (IDS, anti-malware, firewall breaches) is only one of the pieces and to me one of the least important pieces when this type of metric attempts to stands alone and tell the story of the company’s security posture. A technology should be part of a process, not THE process. It is important to know if a technology needs to be recalibrated or changed out, but that should be only a component of the degree of success the specific process is accomplishing.

I have found over the years that many people are confused or have varying different definitions for some very important concepts in our industry.

• What is a security program?
• What is a secure enterprise architecture?
• What is risk? (I find that many ‘security professionals’ do not even understand the true meaning of ‘risk’.)
• What is and how do you define an acceptable risk level?
• What is security governance?

To many people (even security professionals), these concepts are ‘fluffy’, do not have a strict and distinct meaning. So how do you measure something you cannot pin down?

When I work with companies I lay out a very well-defined security program. I find that many companies’ 3 year security plans are as useless as the policies they develop and do not put to use. There is no real reason to develop a 3 year plan unless it lays out very specific steps to accomplish very specific milestones and goals. I define specific metrics to be captured after a process (i.e. vulnerability management program) is defined, implemented, manual steps turn into automated tasks with the right technology, and the process matures so that it happens the same way day in and day out, month in and month out.

I do lay out temporary metrics that should be gathered during the time that the process is implemented and running in production. (Developing and rolling out a full process can take 6 months-3 years depending upon its size and complexity.) These metrics are ‘tests’ to see if we did things correctly by setting up a specific process – they quickly show us our errors so we can tweak the process as necessary. Once the process has all of its kinks worked out and it is something that is repeatable, then the REAL metrics are to be gathered for governance purposes. These real metrics are what is supposed to go up through the chain of command (security administrators, security officer, business unit managers, execs, board) – but the metrics get more broad in nature as they move up this chain. The metrics that go to the security administrators are detail-oriented and move to a more general business meaning as they go up the food chain of the company. The ability to map the detailed metrics to the business oriented metrics is a skill that many people do not possess yet. It requires you to be able to have one foot in the security technical world and a foot in the business world and an understanding of how specific security risks equate to specific business risks. More people in our industry are getting better at this – but this skill is still in the incubating stage.

I find that companies that work in a defined and discipline manner when it comes to any type of business process have a much better chance of rolling out useful metrics. Unfortunately most companies have politics, turf wars, and their own dysfunction that stands in the way of doing this successfully.

In my opinion, defining and implementing successful metrics start with the 3 year plan. If this is just a page long with high-level fluffy ideas of the goals of the security program, it is useless. If you don’t have a clear, detailed plan on accomplishing specific goals, you don’t know exactly what is to be accomplished or measured.

Shon Harris and her team have been writing and publishing articles, white papers and materials on topics vital to the information security industry. Visit LogicalSecurity.com’s Resources section to get free access to its current and archived white papers and obtain valuable information and perspective on security practices.