Dear Security, We Need to Talk About Blocking & Shifting Left

The pressure on software teams to innovate has never been higher, driving companies to adopt continuous delivery and automation. While much has been written about DevOps and Security needing to work closely, in practice few teams are happy with the results. After spending time with companies large and small, both domestically and internationally, I’ve found a few places where teams are commonly getting stuck. 

I have spent most of my career in technology around systems engineering, but the last 10 years have been acutely focused on application and infrastructure automation, and continuous delivery. Security teams have often played an auxiliary role in those automation efforts, but largely I have worked shoulder to shoulder with Developers and Production engineers. Since joining Lacework late last year, I have had the opportunity to interact with a lot more Security Teams than at any point in my career, which has opened my eyes to challenges they are facing to evolve in this rapidly changing landscape.

Wait…You Want Legacy Security Patterns in Modern Cloud???

The movement to “shift-left” by injecting tests and guardrails earlier in the development life-cycle is a proven method to reduce risk and drive down costs of fixing bugs and vulnerabilities that make it into production. As far back as 2016 (an eternity in tech) the State of DevOps report provided data that, “High performers spend 50 percent less time remediating security issues than low performers. By better integrating information security objectives into daily work, teams achieve higher levels of IT performance and build more secure systems.” This isn’t just a best practice, but one could argue that a properly implemented path to production that focuses on shifting-left addresses most of the security challenges we are facing today. The reality is of course you need to focus on both the left and the right, but we’ll touch on that more in a minute.

Contrast the modern cloud approach with legacy security processes and you find dependencies on tools that simply block developers or services via centralized policies, and changing those policies often involves opening support tickets, or even navigating some archaic CAB process, delaying time to market and adding painful friction.

Unfortunately reverberations from that mindset to block persist today. One of the most common questions I still get from security teams is “Does Lacework provide any blocking capabilities?” After digging in with numerous teams I have come to learn this question means lots of things to different companies, but one common use case is around the build process for containers. The idea being that if there are vulnerabilities in the container, will Lacework block the build, or the deployment? This always sparks a lively conversation, but also some serious debate.

The problem is not as simple as, “Does your platform block (yes/no)?” The question whether to block due to vulnerabilities is not binary, because vulnerability and risk are not the same thing. Security teams looking to help their companies shift-left should not be looking for tooling with the capability to block developers. Not only does this miss the mark around outcomes businesses achieve with continuous delivery and automation, but more importantly to do so is to miss an incredible opportunity to dive into one of the most critical aspects elite performers get right, and that is collaboration. The key here is not to focus on blocking, it is focusing on providing the data to make decisions whether or not to block.

Bridging the Organizational Gap with Data

Looking again at the specific security challenges of building and deploying containers, there are many factors to consider, and to know there are vulnerabilities is not enough to make a decision to block. Where in the process do you want to block? Do you want to block developers from building/publishing a container if it has a  vulnerability (not a winning strategy in my experience), or do you want to block a container from being deployed if it has a vulnerability? Do you want to block containers from being deployed into ANY environment, or are there certain environments that need to be treated with greater scrutiny than others (PCI, SOC 2, HIPAA)?

Analyzing the actual vulnerabilities themselves we start to ask how many vulnerabilities are critical, high, medium, low, or info? How many of those vulnerabilities have fixes? Is the service running in a production environment? What is the likelihood of that vulnerability being exploited, and what is the impact to the business if it were to be exploited?

Is it Worth it to Spend Developer Cycles on Every Vulnerability?

Successful implementations of shift-left take into account broader context of the service and how it is deployed on the right. I spoke with a customer recently responsible for addressing container vulnerabilities in a very large cloud environment, and she shared that while their  current solution for container vulnerability scanning alerted the team about vulnerabilities in the build process, without an understanding of where those services were running, she and her team spent all of their time doing “sh***y data science” to calculate the risk. All of this points to challenges that we need more data to make engineering prioritization decisions, and we need far more collaboration across teams.

Developers and production engineers rely on code, automation, and continuous delivery pipelines to get their jobs done. They rely on automated tests to execute and to make decisions off of those tests. They look for robust APIs that they can integrate with and build automated tests from to move fast and efficiently. But fear not, moving fast and efficiently are not at odds with security. The key here in shifting-left is not to slow down or to block, it is to provide access to the data required to make automated decisions around security and risk so we can continue to make those decisions.

A Data Platform to Drive Change

Lacework at its core is a massively scalable, and an elastic data platform. It just so happens that the data we capture, organize and present for our customers is purpose built to help bridge the gap between Security and DevOps so organizations can operate at hyper-speed, without compromising security. But beyond the platform capabilities, Lacework is committed to helping our customers achieve success by reinforcing the best patterns to help teams collaborate together to deliver continuous value without compromising security.

The DevSecOps Journey Ahead

Security teams need to start thinking about blocking very differently. Now is not the time to look to tools to provide blocking as a capability. DevSecOps is more about a culture of collaboration rather than technology. Find a seat at the table with your developers and production engineers, and learn how they are building and releasing their services. Look to provide them with the relevant security data they need to continuously test and assess the risk around deploying those services. Measure not just your own outcomes around security, but also your ability to keep your company delivering new capabilities to your customers with speed and efficiency. In the end we can all achieve our goals, learn, and evolve.


Photo via Ilario Piatti on Unsplash