Five Lessons From Detection & Response Leaders
via the Detection at Scale podcast
Security is constantly evolving, with new attacker groups, ways to break into machines, and clever tools to defend against them. But, becoming talented in security is often the unfortunate result of experiencing incidents first-hand, a true trial-by-fire.
Interviewing security teams is the most insightful way to learn what’s happening right now on the front lines. How are we building effective detection teams? What tools and processes work the best for protecting our organizations? How do we maintain our composure in the stressful world of incident response?
This post details five lessons from 20+ conversations with CISOs, SecOps managers, engineers, and analysts from Netflix, Dropbox, and more.
You can listen back to the full episodes on my Detection at Scale podcast.
Lesson 1: Start with high-quality log data
You can’t protect what you can’t see, so start with a reliable, structured, and expansive data set. A frequent quote is:
“All data is security data because you don’t know where attackers will come from. Application logs, network logs, everything.”
Collecting “high-quality” data means having an authoritative source of truth during an incident. Does it cover our threat model and the most sensitive assets within the organization? Does it add helpful context and increase our confidence during an incident?
The ROI of gathered logs should also be thoughtfully considered. You do not always NEED everything, especially from high-volume sources like network logs, which can greatly increase cost and provide low value for detection or response.
To learn more about logging, check out the post below:
Lesson 2: Detection is becoming an engineering function
A shift is occurring where Detection teams are adopting software engineering patterns to operate effectively at scale.
Detections are moving into code, getting peer-reviewed, checked into version control, and staged before landing in production. This creates a virtuous cycle that promotes a growth mindset in security:
“We should always be striving to get better. Relentless iteration is one of my team’s core tenants. - JJ Agha
By pushing security into code, we gain structure, power, and reliability, enabling compliance, collaborating with various internal teams, and becoming clever and comprehensive in detecting and responding to attacker behaviors.
“You either write code or work with people writing it for you” - Nir Rothenberg
These patterns become the basis for adopting automation, whether in the form of remediation, deployment, or enrichment.
Lesson 3: Automated response is the holy grail
It’s no secret that humans are prone to errors from repetitive work. It leads to burnout and job dissatisfaction and is an inefficient use of security talent:
“If we respond manually more than once, it needs to be automated”
Automation increases the impact of security work and enables teams to focus on the areas where humans are truly needed. There is no replacement for human intuition!
This trend shows that we are evolving from reactive security (responding to incidents) to proactively detecting bad behaviors to taking automated action to resolve issues. The days of ten monitors in a SOC have died, and automated analysis and response will continue to mature SecOps teams into the cloud era.
Practical examples of automation include:
Adding context to an alert (e.g., which team is this person on? what did they do the hour before the alert?).
Pinging users to confirm potential phishing activity.
Isolating machines that have been confirmed to have malware.
Destroying or correcting insecure cloud resources.
As data volumes continue to rise, we will continuously need to automate more!
Lesson 4: Alerts always need a purpose
If alerts are not actionable, they are just noise!
Actionability starts with a question: What behaviors are we protecting against? What’s at risk if this activity happens? What would we do about it? This can be based on internal threat models, compliance frameworks, or prior internal/external attacks.
There should always be a clear next step when an alert is fired, whether automation (mentioned above) or manual intervention.
Alerts should also (obviously) be built for high-fidelity, which gauges quality, and following lessons #1 and #2 above encourages continuous improvement.
Lesson 5: Focus!
The last lesson is to start small and do the basics well.
Optimize for quality over quantity and prioritize the most important assets first. Premature bloat can slow teams down! Let’s keep the agile methodology and move faster than attackers can.
Applying these concepts
These lessons can be applied to new security teams or those looking to evolve their program to be future-proofed.
Change is slow, but the insight above shows how security teams are becoming proactive against the rise of SecOps complexity.
Thank you to all guests who took the time to join our podcast! Check out the latest episode here: