Safeguarding Your Flock Using Anomaly Detection with Lacework’s Polygraph

Imagine you’re a shepherd, tending to an enormous flock. As you look around your vast expanse of healthy sheep, admiring their gleaming white wool, a flash of color catches your eye. You wade through your flock until you come upon its source: a bright blue sheep—an intruder—calmly grazing in your pasture. Something’s not right. You didn’t bring this sheep into your flock, and you don’t know what problems or diseases it may bring. Whether it is an innocent mistake or a malicious prank, courtesy of some miscreants, something must be done. What’s a shepherd to do?

Using this flock of sheep as a model, we can conceptualize how Lacework and its Polygraph technology approaches computer security. Like most security companies, Lacework is based primarily on reactive intrusion detection, because proactive enforcement (i.e., intrusion prevention) entails too high of a risk to business operations when, inevitably, a false positive is raised. In other words, instead of immediately euthanizing any anomalous blue sheep spotted in your flock, Lacework raises an informative alert with full context to your operations and security team—allowing your shepherd to use their judgement in how to respond.

If Lacework were any other security company, its technologies would be based on the tried-and-true methods of rules-based intrusion detection. Rules-based methods are the industry standard in security for good reason: it’s easy to single out and deal with each irregularity, as it is identified. When any irregular “blue sheep” is spotted, it can be quickly addressed by writing a precise new rule: a rule that triggers an alert if, and only if, a blue sheep has infiltrated the flock, with no chance of false alerts.

But mistakes and miscreants abound, in ways that reveal the limitations of rules-based detection. The next day, like a giant puff of cotton candy, a pink sheep may appear in the shepherd’s pasture, requiring a new rule to be written. And the following week, although the shepherd may have proactively written rules for green and purple sheep found in neighboring pastures, they may disappointingly find a zebra-striped sheep in their own pasture—and yet another rule must be written. But these rules will never be enough: the shepherd must anticipate all the other animals, whether woolen alpacas with elongated necks, innocent angora bunnies, or wolves in sheep’s clothing. And the task of crafting precise rules without false positives gets increasingly hard, as rules have to draw ever more subtle distinctions between all the animals. Finally, with sheep as far as the eye can see, the rules must be not merely good, but close to perfect in order to catch intruders without too many false alerts. How is the exasperated shepherd supposed to find time to feed, shear, or groom their flock, and otherwise make a living running the farm?

Luckily, there might be a way for the shepherd to avoid the Sisyphean task of cataloging all the world’s animals. Instead of repeatedly crafting new rules based on an ever-growing selection of unusual sheep (or alpacas, or bunnies, or wolves), what if the shepherd enumerated all the facts about their own animals, and wrote a single rule that specified what a normal sheep should look like in their own expansive flock? Then, an alert could be triggered by any animal that didn’t match this rule. Certainly, this would be dramatically simpler, and easier to maintain as the shepherd experimented with introducing new breeds into their flock. Right?

Unlike other security companies, Lacework performs exactly such anomaly-detection-based intrusion detection: a fundamentally better way to safeguard your flock against intruders.

By successfully providing security based on what is “normal,” Lacework stands out in an industry, where—although attractive in theory—anomaly detection has proven famously difficult to implement in practice. There are four fundamental reasons behind Lacework’s success. First, Lacework provides security for cloud-native software: a scalable, elastic collection of nodes and containers that communicate only by network messages, and are not driven by interactive-user whims but by massive request workloads. Second, the cloud’s online nature allows Lacework, as a flexible SaaS provider, to efficiently get visibility into the full distribution of customer’s activities. Third, Lacework combines information about node activities as well as activities in the network control- and data-planes into abstractions that are stable, even in the face of failures or updates to software or configurations. Fourth, and finally, using patented data-mining and machine-learning techniques, Lacework’s Polygraph is able to learn what constitutes normal behavior for each customer by proprietary algorithmic analysis of those abstractions.

In short—to help security shepherds protect their software sheep—Lacework has cracked the code for how to leverage the massive scale and containerization of cloud computing to give precise, high-signal alerts. By continually observing real-world activities at scale, Lacework can gain full visibility into the long-tail of each customer’s behavior. Thereby, Lacework can simultaneously learn what is and isn’t “normal,” and raise alerts without false positives.

In our analogy, when glancing over their flock, the shepherd might erroneously think all their sheep are all white as snow—even when some are cream-colored or ivory—and receive false alerts, as a result. Because of its comprehensive attention, Lacework would learn those details and not bother the shepherd unnecessarily. At the same time, Lacework can be trusted to raise an alert on any truly anomalous animal, such as a wolf in sheep’s clothing.

Lacework’s customers testify to the outstanding success of its behavioral anomaly detection, to the point where Lacework usually ranks #1 in customer satisfaction across comparable security technologies. In particular, customers love that Lacework raises very few alerts: usually, they see no priority alerts in a given day, or perhaps just one. Simultaneously, when technologies are empirically evaluated by customers—e.g., by concurrently deploying competing security technologies and performing a pentest—Lacework’s performance matches or improves on that of the competition, without suffering from their rate of positives.

Moreover, by learning a baseline of what should be considered normal, Lacework extracts information that is valuable across the entire devsecops spectrum. Many of Lacework’s most enthusiastic users are not security engineers, but rather developers or operators that rely on Lacework’s Polygraph to summarize the normal set of activities in their cloud. These devops roles value that Lacework raises alerts any time that “normal” is disrupted, even when that disruption is due to a failed release, rolling update, or confirmation change.

The future is now: cloud-native software has fundamentally changed how software is developed and deployed, and Lacework has shown how, for cloud software, security can be enforced using fundamentally improved methods. By relying on scalable, data-driven techniques and ubiquitous online monitoring, security need no longer be based on a never-ending list of rules that enumerate known bad activity. Rather, security can be based on learning the characteristics of normal, healthy operations of cloud-native software—with alerts raised whenever operations diverge from what is normal—and this can be done in a manner that benefits all of devsecops, more easily, at reduced cost, and without loss of accuracy compared to traditional security.

So go back and tend to your flock. Lacework will watch out for the miscreants, as well as any mistakes.