Don’t just shift left, level up: Building a modern cyber defense program

Within the security community of late, the focus has been on “shifting left”, and while that has merit, it is somewhat myopic missing some of the realities of defense in practice. Instead, I propose a simple framework to help guide initiatives that will “level up” defenses and greatly improve security postures wholistically. Some license is taken in terminology in order to keep things simple, memorable, and applicable.

We start with the three basic pillars of development, design, and detection. To be more specific, secure development (or DevSecOps), secure architectural design (such as zero-trust architectures & security reference architectures) and lastly, modern detection and incident response. These three pillars are built on the foundation of automation and doing “everything as code”, including Infrastructure as Code (IaC), Detection as Code (DaC), and Security Orchestration Automation and Response.

Why is automation the foundation supporting the three pillars? Because manual security processes are prone to human error, such as accidental misconfigurations, drift from secure baselines, or in the world of detection--distractions and alert fatigue. Manual processes also don’t scale, limiting the ability to have a consistent security posture that is centrally managed and monitored.

Note that each of the three disciplines applies automation and “as code” in different ways, but the most mature teams are aggressively moving to these programmatic approaches. Unfortunately when organizations fall short in any one area, even the most mighty houses can fall.

Development

In a previous article, I emphasized that end-to-end visibility starts at development time. Where we once focused our security efforts on securing single monolithic applications and servers, the new modern app is spread across containers, microservices, and different cloud providers, and unless you design these complex apps with security from the start, the battle may be already lost. In the military, they speak of “moving left of bang”. The guiding principle is, it's better to detect adversarial intentions early than respond to malicious actions after the damage is done.

Fortunately, developers fix problems, and they do it programmatically. We now have tools to do security validation early on in the development lifecycle. This is especially true with the open-source community rallying to solve the hard problems of supply-chain risk, software vulnerabilities, and instrumenting telemetry at the lowest levels of our software. I described this previously as “the ultimate race condition” in which developers must create tooling to secure critical applications, continuously validate they are secure and operating as designed, and incorporate these functions into the CI/CD pipeline and are part of the build and run-time processes. This reduces the security risk, but everything comes at a cost. Having these additional gates and checks does create overhead that can slow things down, and that then becomes a business risk.

Creative and bleeding-edge development drives business revenue and is the reason companies can innovate at high speed. From a business perspective, security is a necessary evil. And although cyber doesn’t add to the bottom line, by leveling up cyber defenses, organizations can reduce the risk of cyberattacks and their associated costs, including financial losses, reputational damage, and legal liabilities. It also helps ensure that critical business functions can continue in the event of a security incident.

Positioning secure development as an investment, and not a cost center helps reframe the conversation. As security practitioners, we need to be able to articulate the benefit that comes from secure design processes and DevSecOps. Here are four principles I advise security professionals to keep them aligned with business objectives.

• Understand the “why” behind your business. Your role is to support the business. Sometimes that means accepting risks.

• Don’t make enemies with the developers. Help them work smarter not harder to ease the burden of additional requirements.

• Stop saying “no,” and start explaining “how.” Flexing your “leet speak” is not helpful to those making business decisions.

• Remember that security can be part of the “sale”. Help the business frame security strengths as a value-add that customers benefit from. Security is an investment and customers are willing to pay a premium for secure products.

To dive deeper into how to make your development shop an “elite performing team”, read how to accelerate and secure your SDLC with DevSecOps. It’s not enough to follow these secure development practices once. They must be part of the development process at the deepest level.

Implement tools that automate static and dynamic application security testing (SAST/DAST), software composition analysis (SCA) and also validate API and container security. This achieves the desired “shifted left” results. Once that is achieved, this secure software needs to be pushed into production securely. This is best done through infrastructure as code (IAC) and leads us into the next related pilar of secure design and adopting proven security reference architectures.

Design

Remember shifting left is just one leg of a three-legged stool. Once you have secure code, it needs to be securely deployed and monitored with automation. Designing the architecture and the infrastructure that will run applications is the next consideration.

I’m using the term “design” here not in the sense of software design, as that is covered under development. At the most granular level, the design pillar incorporates modern application architectures. At a higher level, it represents the myriad of tools running in a modern enterprise security stack. Note there is significant overlap here, as well as overlap between design and development. In fact the lines between software and infrastructure/architecture continue to blur as most everything is done through code at some level.

Security Posture Management tools have come a long way, allowing policies to be created, and when configurations deviate or vulnerabilities get pushed into production, alarms can be raised. I’m most excited about the advancements of open-source Cloud Security Posture Management (CSPM) tools like CloudQuery and SteamPipe. Because these solutions are modular and open, they easily fit into a Multi-Cloud Compliance as a Code strategy. Here at Sumo Logic, we’ve been experimenting with leveraging AI/ML to intelligently ingest vast inventory data and their respective configurations, and then provide remediation steps down to the actual commands an administrator should use to address vulnerabilities and security findings.

Having a centralized platform that spans development, operational and security use cases is one of the reasons customers love the Sumo Logic solution. The number of required solutions to run a secure enterprise continues to grow, and having a tool to cross correlate and allow collaboration from different business units is paramount.

Distinct and purpose-built security solutions provided by vendors will continue to flourish, but it’s necessary to have some blueprint on how everything should be wired together. To run a successful security machine, every gear and sprocket has to be in place. Integration across systems is key. Here we can lean on what is known as Security Reference Architectures and a Zero Trust Architecture.

As NIST defines it, Zero trust (ZT) provides a collection of concepts and ideas designed to minimize uncertainty in enforcing accurate, least privilege per-request access decisions in information systems and services in the face of a network viewed as compromised. Zero trust architecture (ZTA) encompasses component relationships, workflow planning, and access policies. Therefore, a zero trust enterprise is the network infrastructure (physical and virtual) and operational policies that are in place for an enterprise as a product of a zero trust architecture plan.

According to Forrester Research, a ZT architecture "abolishes the idea of a trusted network inside a defined corporate perimeter. ZT mandates that enterprises create micro-perimeters of control around their sensitive data assets to gain visibility into how they use data across their ecosystem to win, serve, and retain customers."

In a related effort, Microsoft, with its Cybersecurity Reference Architecture (MCRA), has done a lot to bring security architecture and design principles into a single cohesive set of blueprints. Their reference architecture is more vendor (Microsoft specifically) focused but describes how customers should enable zero trust user access, security operations, attack chain coverage, cloud providers security controls by integrating across vendor solutions and 3rd party apps.

Good security, compliance and identity solutions require deep integration across ALL your other solutions. When building a wall around your castle, having a gap in any area negates all the other effort spent. A recent article by Allie Mellen, a Senior Analyst at Forrester, broke this down even further by separating “attack surface” from “detection surface”. We can build castle walls, and motes to protect critical assets, but what are all the detections, cameras, trip-wires and virtual guards we have in place to ensure these defenses are working properly?

When looking at all the interdependencies, a comprehensive security reference architecture can be head-splitting. Here at Sumo Logic, we distilled this down into high-level solution verticals in a Modern Enterprise Security Architecture.

When done correctly, your organization will not only have “shifted left” but it will also have “leveled up”. To level up cyber defenses means enhancing an organization's cybersecurity posture and increasing its resilience against cyber threats. Cyber resilience is achieved as this involves a combination of technological solutions, policies, and procedures all designed to protect an organization's critical assets. A next-generation SIEM (security incident and event management) solution lies at the heart of these integrations.

When you are evaluating new solutions, do not view them in isolation. Increase the scope of your proof-of-concept testing to include integration across the stack into your SIEM, SOAR and identity solutions. Your engineers and analysts will thank you when they are handed a new shiny thing to integrate with and it actually works, especially when it’s not one more tool they have to log into on every shift. Bonus points if you can incorporate infrastructure as code principals where all configurations can be applied programmatically.

Detection and response

When everything above fails, and attackers still breach your environment (and they will), you need a modern detection strategy in place. As cyber defenders, we have to assume that software will be vulnerable. We have to assume misconfigurations will happen. We must have a “breach mindset”. If it's truly a matter of “when” not “if”, our software should have the required hooks embedded at the lowest levels to provide the observability and telemetry we’ll need when things go haywire.

This for me is the most exciting area. Huge advancements are taking place in the ability to dynamically track and detect attackers' activity, providing analysts and feeding automation leads to high-fidelity alerts. OpenTelemetry standardizes the collection of MELT telemetry (Metrics, Events, Logs and Traces). Sumo Logic has greatly contributed to OTel as it provides the data that feeds our SIEM, UEBA and SOAR solution. Having these three solutions in a single security platform, underpinned by a single security datalake has been a game changer for our customers.

Why is SIEM + UEBA + SOAR a game changer? It’s a 1-2-3 punch because first, with the advent of true user and entity behavior analytics (UEBA) capabilities that baseline entity behavior to identify unusual patterns, alerts become more high fidelity. These UEBA detections leverage machine learning algorithms and statistical analysis models to establish normal behavior or patterns per user or system in order to detect anomalies. Unfortunately, UEBA solutions alone have been extremely noisy and customers had difficulty baking in the “normal” baselines before good alerts were generated. The juice of having a separate UEBA solution just wasn’t worth the squeeze. Things are different now. Case in point, advanced UEBA directions are now natively part of the Sumo Logic detection strategy, as is our new automation service borrowed from our SOAR technology.

Now that these alerts are correlated within the SIEM alongside other proven detection rule types—at scale across all users dynamically—we finally have extremely actionable insights that may be a cluster of multiple alert types. With these insights, actions can be taken immediately and in many cases automated.

There are some pitfalls to automating cyber defenses. Historically, the garbage in, garbage out dilemma has been a major roadblock for automating security operations. The last thing your security team needs is to be blamed for conducting a DoS (Denial of Service) against critical systems because of a false-positive action or errant remediation that blocked paying customers from your services or prevented the company from operating.

Having advanced playbooks that can pause for analyst interventions and validation is a good way to reduce some of this risk. Many of the hundreds of out of the box low-code playbooks we provide have such human actions part of the response and remediation flow.

Today we have very powerful ways to classify adversarial behavior and indicators of compromise. A modern cloud SIEM should be able to generate alerts, combine these alerts with those provided by point solutions (EDR, IDS, WAF etc) and then overlay these events into a single attack timeline. This timeline should also include the MITRE ATT&CK adversary tactics and techniques.

Triaging isolated alerts does not scale and is extremely ineffective. As aptly put by Bill Crowell, former NSA Deputy Director, "Cyberdefense is about having an integrated set of tools that work together to prevent attacks, but the industry now has a thousand points of light and no illumination."

When considering a modern SIEM+UEBA+SOAR platform, ask yourself the following:

• Does it include a broad set of automations and integrations out-of-the-box, with little or no additional charges to deploy automations?

• Does it operate in an entity-centric way, automatically correlating observable events to entities (e.g., users, systems, IPs, etc).

• Does it include multiple detection capabilities within its rules engine? For example, ask vendors to demonstrate each of the following detection techniques with out-of-the-box rules:

  • First Seen rules: Used to detect any suspicious, never seen before behavior based on learned baseline. It’s critical that the first seen rules engine allows for both dynamic entity-based baselines or enterprise-wide baselines.
    
    Entity based example: Bob always does things X, while Alice does things Y. Alert me when either Bob or Alice does something abnormal for them based on a previous learned baseline.
    
    Enterprise or global example: All users within the organization are expected to do things Z, however when any new behavior is seen across ALL users, create an alert.
    

  • Outlier rules: Used to detect patterns in data that would otherwise not be considered suspicious and require dynamic baselines learned over time that are unique to individual entities.
    Example: Bob usually uploads 10MB a day to external sites, but this week his outbound traffic was 100x his normal indicating possible data exfiltration or a compromised account.
    

  • Chain rules: Used to detect when two or more distinct events occur in a given time window. These events may be from unrelated data sources.
    Example: Five failed login attempts followed by one success in under one hour.

That’s a lot of words, but what does this look like applied to an actual breach scenario we see within our customer environments? Here is an example of a correlated Insight following a successful phishing attack.

Of course, all of this requires the stars to be aligned and each of your detections fire appropriately. Multiple detection strategies help mitigate false negative rules, however. Adversaries should be tripping detection wires as they move laterally and do attacker things.

Case in point above, an astute observer would notice the lack of an EDR endpoint alert. Ideally, this type of breach would have fired such an alert when malicious binaries were observed on the host. However, that is not always the case, and even when one point solution fails, cross-correlation across all data sources makes up the difference. Again, this is a testament to the power of entity-centric correlation only a SIEM can provide.

Let’s underscore the claim that was made where automation and “everything as code” is the foundation of all three pillars. How does it apply to detection and response? In the example above, note that from beginning to end--the detections, enrichments and notifications, were all automated and done “as code”. Security engineers should also be driving towards autonomous SOCs that integrate defense into automated playbooks for both enrichments and remediations.

Katie Teitler had a great piece titled “IT Spend in 2023: Why Automation Will be the Hero”. In it she states that given the state of cybersecurity staffing, the current economic climate, and greater pressure on security teams, a survey of 213 cybersecurity executives in the U.S. found that automation was top of mind for efficiency, efficacy, and cost cutting measures. And 86% of cybersecurity leaders are prioritizing automating threat detection and response, and 84% are prioritizing integrating and automating cybersecurity capabilities with new and existing technologies.