How To Get a Handle on the Log4j Issue in Your Environment
Editor’s Note: Log4j is an unfolding situation that we are closely monitoring. We will be updating our blog on a regular basis so check back for updates.
The vulnerability with the Log4j logging library has been keeping IT teams up at night since late last week. We know when a zero day event occurs like this it creates understandable panic and we’re here to help. Naturally when an issue like this comes up, your sole focus should be on responding to the issue. You need to reduce the risk to your organization as quickly as possible.
To that end, here’s a cheat sheet to help you plan that work and to help make sure that nothing has fallen through the cracks.
What is Log4j?
Log4j is an open source library for applications written in Java. It helps developers record what’s happening in their applications. These recorded logs help make sure that things are working smoothly and they provide critical information when things aren’t going as planned.
Where is Log4j used?
Log4j has been around for years and is currently in its second major version. This library is so popular in the Java community that it is basically the standard for logging.
This means that it’s used in most applications that are written in Java. The problem is that most users are unaware and don’t need to be aware of what language their applications were written in.That’s why you’re seeing reports from various services and vendors about updates being required. Log4j is part of the plumbing that runs most Java applications.
What happened?
On Thursday, 09-Dec-2021, the team of volunteers behind Log4j announced a serious vulnerability in the library. There are two CVE (common vulnerabilities and exposures) identifiers assigned to this issue, CVE-2021-44228 and CVE-2021-45046.
This vulnerability had the two things that we never want to see in a security issue, let alone see in the same security issues because it’s easy to take advantage of and gives attackers the ability to run their code on your systems.
Officially, the severity of this issue is rated 10 out of 10. “That’s not good” is a massive understatement. In simple terms, this means it’s worth stopping normal operational and security work and addressing this issue immediately.
What do I need to do?
Responding to incidents like this is always a challenge. Because log4j is part of the plumbing of some applications, getting a handle on the scope of your exposure is very difficult.
Here’s the general process you’ll should be following;
- Monitor production for exploit attempts
- Look for all installations of log4j in your environment
- Prioritize that list of affected systems
- Working through that list, either turn off the affected feature or upgrade to the latest log4j version
While that list seems straightforward, it presents a number of challenges.
A number of researchers and vendors in the community have reported widespread attacks related to this issue. Making matters worse, attackers are using a number of different techniques. This means you’re not looking for one but several different types of attacks on your network. This post from Lacework Labs highlights some of the techniques we’ve already seen in the wild.
Cybercriminals know they have a window of opportunity here as teams rush to mitigate the issue. That means that you and your teams have to balance fixing the problem while monitoring production for active attacks.
This is a really hard balance to strike, but it’s a critical one. You don’t want to close one gap just as attackers gain a foothold through another. In production, you should be focusing your efforts on looking for anomalies in your activity data. That will help spot the new techniques as they are deployed by cybercriminals. In the end, security controls will only buy you so much time to fix the root cause, the version of log4j in use. Finding the vulnerable systems is the biggest hurdle. You can’t fix what you don’t know about.
Vendors with affected products will be notifying customers. That will happen either through direct contact or their blogs so make sure to check in with your vendors to understand how to mitigate the issue for those products. The community has also published several scanners to help determine if specific systems are affected that might help you out. If you’re thinking, “This is a lot of manual effort.” You are correct. Sometimes there’s no way around it. Each set of systems is going to need to be checked for vulnerable applications and a specific plan made to address each occurrence detected.
Once you know the scope of the work, it’s time to walk through the dependencies in your network. Some systems will be simple fixes. You can turn off the impacted feature of log4j or quickly patch while others will require coordination between teams and testing of the remediation plan. Hopefully you’ll get to the point where you have a grasp of what needs to be done. The next step is to order that work to reduce the risk to your organization as quickly as possible. High value systems should be done first, working your way through the long tail of the list. It would be nice if there was a faster way, but there isn’t.
The key thing is to remember the ease with which attackers are taking advantage of this vulnerability. We’ve seen multiple reports of attackers taking an opportunistic approach with this vulnerability. They are scanning systems at will and attacking any they find vulnerable.
Remember that risk is the combination of the likelihood of an event occurring and the possible impact from that event. We do know that attackers are actively hunting for victims which means there’s a strong likelihood of your systems being attacked. If that attack is successful, the attacker gains access to your environment and can run their code. That could have a significant impact on your business. This is one of those times when downtime and other disruptions are usually an acceptable trade off in order to reduce the risk to the business.
I’m a Lacework customer, what do I need to know?
The Lacework Customer Success team has documentation available to help you understand how to use the Lacework platform to help uncover any vulnerable systems and aid with any investigations related to log4j. With the Lacework Polygraph technology, you can identify anomalous behavior that may be indicative of an attacker trying to exploit vulnerable systems. This is important as you actively patch your environment and need to pay extra attention to any vulnerable services running in production. Additionally, you can leverage the Polygraph visualizations to look at New Connection Events from Java applications. As with any major incident, this documentation will be updated as new information becomes available.
Lacework Labs will be monitoring for post-exploit activity, including historical data. We will provide specific recommendations to customers if a compromise is detected.
What’s next?
The situation is likely to change quickly over the next few days. Please continue to monitor the updates from your vendors and work through the list of impacted systems.
It’s exhausting work but the critical nature of this vulnerability means it can’t wait.
Stay tuned to the Lacework blog and social media handles (we’re @Lacework on Twitter) for more as this situation develops.
Copyright 2021 Lacework Inc. All rights reserved.
