Lack of effectiveness metrics and orchestration/automation top list of security operations frustrations

The more things change, the more they stay the same. SANS recently released its 2018 Security Operations Survey, and we continue to see the same barriers to SOC performance and effectiveness rise to the top.

talent-retention--670x310Limited pool of skilled security operations talent
metricssmUnclear or undefined SOC metrics
IntegrationLack of orchestration/automation, integrated toolsets and processes/playbooks

It’s easy to see why security operations teams continually cite these three challenges, as they are tightly intertwined. They represent the two main options facing enterprise and MSSP security teams when it comes to efficiency and effectiveness – either throw more bodies at the problem or figure out how to adjust processes and other resources to make more of existing staff.

The journey to solving the talent shortage is long, winding and complex. So for the purposes of this post we will take a closer look at what SANS uncovered relative to the other two challenges – metrics and integration – as those can be more practically addressed in the near term.


The Security Operations Metrics Conundrum

Security teams are constantly asking for more budget for resources to improve their day-to-day operations. Yet, SANS found that only 54% of SOCs claim they provide metrics that can be used in reports and dashboards to track the ongoing status and effectiveness of their performance.

SANS metrics

Knowing that securing more funding for additional hires and new technologies rests on providing data and proof – why aren’t more SOCs diligently tracking their performance?For starters, many security operations teams say the reporting they are able to provide requires a significant amount of manual work to pull together. For SOCs already feeling the weight of too many alerts to investigate and too many tools to manage, finding the time to do reporting seems all but impossible.

Manual metrics

Second, the metrics that can truly illuminate the health of a SOCs incident response processes – such as mean time to detect (MTTD) and mean time to respond (MTTR) – can be complex to determine because they require a thorough understanding of security incidents. For many teams, this sets up a huge catch-22; they don’t have the holistic view needed to measure and demonstrate MTTD/MTTR so they can’t make a data-driven case to management that resources are needed to maximize performance against those metrics.This leads us directly to the next challenge. See? We told you they were intertwined.

Not Enough Integration Means Not Enough Insight

number of respondents using technology

Every SOC is unique, using its own blend of off-the-shelf and homegrown security tools for the prevention against and detection of cyberthreats. Turning these disparate tools – which number more than 20 for 2/3 of respondents in the SANS survey – into a cohesive ecosystem is laborious, takes significant resources and requires a tremendous amount of care and feeding.

The real challenge for security operations teams, however, is the level of work and time required to make sense of the data coming from all of these different security tools. It’s time consuming to jump between screens to gather details around a single alert, and  analysts simply don’t have the time investigate all the alerts they are barraged with each day, let alone build the deep insight needed to create a full storyline around a particular alert or threat.

This is why we are seeing automation and orchestration show up near the top of the list of SOC needs.


Security orchestration and automation can not only enable the integration of disparate tools for more streamlined management but with deftly applied automation, data from these disparate sources can be better correlated and enriched. From this, analysts get deeper insight into alerts and security events that can ultimately speed up investigation, triage and incident response. And because security orchestration and automation platforms can serve as a centralized hub across the SOC – bringing together people, processes/playbooks and technologies – many include robust dashboard and reporting capabilities that address the metrics challenges cited previously.

Other Takeaways from SANS

Incident response stays in the family.

The majority of security operations respondents say incident response is a fully integrated function of their SOC. While outsourcing of security services is common, the survey doesn’t indicate a high level of adoption of external IR services.




SIEMs are still the primary tool for correlation
correlation SIEM

Most SOCs cite SIEM as their primary correlation tool for security event data. However, SIEM alerts came in third on the list of indicators that trigger a response. And the vast majority of SOCs still rely on manual 


soc activities

Pentesting and threat research are outsourced most frequently.

More than 80% of SOCs do some level of penetration testing and threat research internally, but these externally-focused activities are the ones most likely to at least be partially outsourced.