Recently I took my 4-year-old daughter to the doctor’s office for a run-of-the-mill ear infection.

Generally our wait time with this doctor is short. Unfortunately, this was a Saturday and apparently every child within a 25-mile radius of the office was sick, so the delay was exceedingly long.

After more than an hour, we finally made it into an examination room. Of course, the actual visit took less than five minutes. A quick look is all it took for the doctor to write a script, and off I went to the pharmacy. A wonderful way to spend your Saturday, let me tell you.

Now, what in the world does this have to do with the mean time to detect (MTTD) and mean time to respond (MTTR), two of the most important metrics emanating from your security operations center (SOC) because they describe the time spent identifying and extinguishing a threat.

If you look at the personnel makeup of your doctor’s office, you can draw some parallels to your typical SOC:

Receptionist: Manages intake of patients, pulls the patient file, handles payment.

Nurse/physician assistant: Moves patient from the waiting room to the exam room, takes some initial diagnostics (height, weight, temperature, blood pressure) and makes notes for the doctor.

Doctor: Conducts the exam, makes a diagnosis, and, if needed, writes a prescription

Now let’s model, at a very high level, a typical SOC:

Security Control(s): Identifies a suspicious action/behavior/file, creates an alert which is consumed by a SIEM

Tier 1 Analyst: Review alerts, perform initial triage, escalates alerts for investigation by Tier 2 or Tier 3 analyst

Tier 2 or Tier 3 analyst: perform detailed investigation and takes remediation actions to mitigate security risk

You can probably see where I am going here:

  • Receptionist = Security Control
  • Nurse/PA = Tier 1 analyst
  • Doctor = Tier 2 or 3 analyst

MTTD is the amount of time it takes to discover a potential security incident. In a doctor’s office, this would equate to the time between you entering the office until the receptionist checks you in. Normally this is very short, although I have certainly spent my fair share of time staring at an empty receptionist’s window. 

During the time between entering the office and getting checked in, I could wreak all kinds of havoc: rifling through the magazines, changing the TV channel, raiding the mints, just to name a few. Even after being checked in, though, the receptionist moves on to the next patient, knowing I am still waiting but having no time to deal with me. This gives me another opportunity to cause mischief. In fact, up until the time I am handed off to the nurse, I am still able to run free doing whatever I like. 

The same holds true for our trusted security controls. Even a middling attacker has a good chance of evading detection for a long period. According to a recent Ponemon report, attackers explored compromised networks untrammeled for an average of seven months before being detected the extent of an entire baseball season. And that’s not even the worst part. Once your security controls detect the issue and generate an alert, it moves on. The control did its job, found something odd and noted it, but that is where its job ends. If no one ever looks at the alert, oh well, not its problem.

Mean Time to Detect

So how can a doctor’s office, and a SOC, reduce MTTD? For the doctor, it’s pretty simple: Add a bell or ringer that goes off every time a patient walks into the office. Assuming the receptionist is within earshot, there is a safe assumption that the patient will be promptly checked in.

For a typical SOC, there are a number of principles you should follow to help drive down MTTD. 

1) Ensure your security controls are deployed and properly configured. One switch on or off can make the difference between identifying a threat early or having an attacker run through your environment like a kid in a candy store. Understanding proper configuration is easier said than done, but you can tilt the odds in your favor by building a smart security stack with complementary not overlapping capabilities. 

2) There is nothing more annoying than seeing multiple alerts for the same potential security issue. By adopting complementary technologies with limited overlap in capability, you can dramatically reduce the number of redundant alerts, giving your team a better chance of not missing a critical issue. 

3) With complementary capabilities in place, you are likelier to detect an attacker’s behavior at some point in the kill chain. For example, if you fail to pick up the initial exploitation of a vulnerability, you can rely on another tool to sniff out command-and-control traffic. In short, structure your security approach to detect malicious activities across the kill chain to decrease the chance an attacker can move unrestricted in your environment. 

Mean Time to Respond

Now, let’s talk about MTTR. Back at the doctor’s office, a patient is checked in (detected), and now he or she waits to be seen. This can vary from a few minutes to hours (we’ve all been there). The nature of how a doctor is paid drives up their MTTR. For example, most doctors in the United States are paid by procedure and not by time spent with a patient. Thus with every patient having different needs, the MTTR can vary widely. If every patient before you has an ear infection, you’d be quickly diagnosed, However, if every patient requires in-office tests, be prepared to camp out in the waiting room. 

SOCs face a very similar dilemma, but on a much larger scale. Even with a finely tuned security stack, the number of alerts generated daily will far exceed the capacity of the team to respond. As such, most SOCs adopt a “tiered” or “teams” approach to how their personnel is constructed. In a tiered approach, a Tier 1 analyst will perform initial triage (like a nurse would in a doctor’s office), while Tier 2 and or 3 analysts (the doctors) perform full investigations and remediation. 

The problem remains, though, that even with this approach, there are simply too many alerts to manually perform triage and investigations. So, how can a doctor’s office, and a typical SOC, drive down MTTR? Many doctor’s offices now ask patients to fill out their intake forms before they visit the office. Assuming the patients properly fill them out, the doctors gain visibility into their caseload for the next day (or days) in advance and can take appropriate steps. If several complex cases are on the docket, they may notify other scheduled patients ahead of time and request rescheduling or, if possible, add staff to handle the caseload. 

For SOCs the answer is even easier: orchestration and automation. Now more than ever, security teams need to adopt automation to keep pace with their alert flow. With proper orchestration and automation, security teams increase their capacity by magnitudes. 

With recent surveys showing analysts burn an average of 15 minutes every hour on false positives, adopting an automated approach to alert triage, which is where false positives are typically identified, could increase your SOC capacity by 25%. Today many organizations are adopting security orchestration, automation, and response (SOAR) solutions to achieve this very goal. With a SOAR integrated into your security architecture, alerts will flow automatically into a process where they can be analyzed and actioned upon with little to no human intervention. An appropriately deployed SOAR solution will not only help with the false-positive problem that drives up MTTR, but will also speed the investigation once the full investigation begins. By building workflows into the SOAR, also known as playbooks or runbooks, analysts tasked with performing the full investigations will have all the data they need at their fingertips with minimal effort. 

If MTTD and MTTR are keeping you up at night, consider these recommendations and turn the tables on the attackers. You’ll probably still have a wait ahead of you the next time you visit the doctor, but at least you won’t be sitting there worrying about your SOC.

Steve Salinas is director of product marketing at Siemplify.