Blog

A very high profile security issue of the log4j library, registered as CVE-2021-44228, dubbed as “log4shell” keeps admins on their tows to get their systems sorted out and updated as soon as possible.

This article is meant to answer some of the questions we have been getting regarding log4j and mainly how it impacts high performance clusters.

To find the remedy, we need to understand the problem. Log4j is a java library. Two important things to notice here:

1. it is for JAVA applications, so only java applications are directly affected by this issue and
2. Log4j is a library, not an application!

Log4j versions 2 to 2.15 have a vulnerability that allows to execute code on the machine where the application using log4j is running. Log4j is used to log information within an application and write log files. The vulnerability dubbed log4shell is a bug, where writing some crafted log entries could lead to commands being executed on the machine where the software using log4j is running, with the user rights of the user that’s running that application. The reason why this is such a high profile security issue is, that under certain circumstances, it is extremely easy to inject such crafted code into the log message. For example apple was initially targeted by changing the name of an iphone to a crafted string that would then hit this vulnerability. So in other words, under certain situations, it is extremely easy to exploit this bug, and that makes it very dangerous, because attackers can act quick to leverage the vulnerability.

This issue affects servers that are publishing services, such as for example web services or databases etc. to users who are not authorized to login to the server and run their own software on the server. For a user that already has ssh access to a shell of a server, there is very little to gain by exploiting a log4shell affected application. The only danger that log4shell poses in such scenarios is, if a tool running with higher privileges than what the user has can be exploited. Such tools could be monitoring tools that must run as root in order to access the hardware that they are monitoring for example.

So does this only affects web-services written in java? No, because user input could also be fed into an application by modifying data that is later processed by a vulnerable application. Let’s say, you have a web-application that has a search field. Your Web application is written in something other than java, so you should be safe. But lets suppose you store each search query to a database, because later on you want to analyze what people are searching most on your application in order to improve the user experience in the future. To analyze the data, you use another software that will read all these search queries from the database and then process them in one way or another. Unfortunately that application is written in java and uses log4j to write debug-logs for the developers to improve the inner workings of the application. Now suddenly you have user input hitting a java application that is not directly exposed to the outside.

To asses if a system is vulnerable through the log4shell issue, we have to first of all find all applications on our system that use log4j between version 2 and 2.15. The challenge here is that Java applications are distributed as jar archives which may contain also all the dependencies of that application, so all libraries. A jar file can also contain more jar files which in turn contain software and its dependencies. This means we have to really iterate through all levels of a jar archive in order to collect all the components provided with a java program. Because log4j is a library, it may very well hide in a jar several levels deep.

this is for Linux systems only, but the main tool we use here (syft) is also available for windows and the other tools have probably alternatives available on windows as well. Our approach to finding most or hopefully all occurrences of log4j on a system is the following:

1. we use find to locate all .jar, .war, and .ear files on a system.
2. we then use an open source tool called syft to analyze the contents of these files. syft can output detailed information about a software package in json format.
3. we then use the command line json processor “jq” to query the output of syft to find occurrences of log4j and list them together with the name, version number and path of the software package that contains the library.

both syft and also jq are not usually part of a standard linux installation, so you may have to install those tools first. JQ is usually available via the package manager of the distribution, so a simple
yum install jq
or
apt install jq
will most probably do the job.

syft provides a one-liner on the https://github.com/anchore/syft github page that will install it. If your system does not have internet access, you can also download the respective rpm or deb package for your distribution and install that instead. Just look at the releases page.

Once these two dependencies are installed, you can use this linux command (enter it all at once, it is basically a one-liner split into multiple lines for better readability) to scan all the files on your system with the before mentioned method. It will also create a file called /tmp/jarfiles in case you want to run some further analysis on all jar files you have found in the future. It also saves all the output to /tmp/log4j, just in case it finds more than your screen buffer can scroll back :). Keep in mind that this command can take very long, depending on how much data you have on your sytem. So you might want to run this in a “screen” or “tmux” session.

find / \
-regex '.*\.\(war\|jar\|ear\)' \
-exec /bin/bash -c '\
echo "{}" >> /tmp/javafiles;\
syft packages file:"{}" -o json 2>/dev/null | \
jq -r ".artifacts[] | \
select(.name|test(\"^log4j\")) | \
[.name, .version, .locations[].path] | \
@tsv"' \
\; | \
tee -a /tmp/log4j

if you want to only scan a specific folder, replace the first “/” after the “find” command with the path to the desired folder.

the list generated by this command will show you probably all occurrences of log4j on your system. it also includes versions below 2 which under some more complex circumstances could be dangerous but most probably aren’t and it includes log4j-over-slf4j which by itself is not an issue unless an affected log4j version is found in the same package.

First of all, check if there is an update for the software that’s using log4j that fixes the vulnerability (usually by using a newer than 2.15 version of log4j). If there is and if you can, upgrade to the newer version.

If however you can’t upgrade, you need to analyze, if this vulnerability is even an issue in this specific software and use case. If the software in question is a solver application for some simulations that you are running on a cluster which is only used by yourself and where you aren’t afraid of a privilege escalation carried out by a very competent co-worker who also has access to the system, it is probably safe to leave the software as is and continue using it. If all your system users have shell access anyway (such as on an HPC cluster) and there are no exposed services other than ssh, make sure the application in question is not run as root and you are most likely fine too. However, if you can’t say for sure that it won’t be an issue, you can either try to google for a workaround but most probably what you should do then is just to remove the software from your system until there is an update. In the case of services, you can also stop the service and disable it until there is an update.

One very popular tool that we have seen mentioned by many people seems to be the log4j_checker_beta by rubo77 on github. It certainly is a start but this tool is incomplete in some ways which may give a false sense of security in some cases. First of all, it uses “locate” to find java files on the target system. Locate is a file indexing tool on Linux which comes with a tool called “updatedb” to regularly scan the system for new files and add their names to a database. Later, locate can query the database for a filename or expression to find files on the system quickly. The log4j_checker_beta tool warns, that your locatedb might be outdated and that you should update it prior to running the tool, however, if you don’t have the mlocate package installed and there is no locate program available, it does not warn about anything. Instead, it falls back to using “find”, a tool that simply crawls all your directories and searches for files. This is much slower but much more complete way of finding a file. Because find is so much slower, the script only searches for java files in certain directories, where they often reside on Linux, such as “var”, “bin” and so on. But as soon as you have other paths where you installed software for one reason or another, it won’t find it unless you have an up to date locatedb, and sadly it does not tell you if you don’t have that.

Also mlocate can be quite resource hungry when updating its database, so many admins remove it from systems that provide access to large amounts of data (this includes us in our cluster and storage installations)

Another shortcoming of this script is, that it only looks into jar files one level down, so you might miss log4j if it is embedded deeper down inside a jar file.

sometimes you see short “find” one-liners that just look for any file with “log4j” in its name. This is sadly not very useful, as it will not show any libraries contained in application’s jar files and that’s probably the majority of occurrences of log4j.

another misconception is, that if you don’t have java or log4j installed via a packet manager or if the java command is not a known command on your Linux installation, you are safe. Often times, java applications come with their own java runtime packaged together with the application. This means that even if there is no java in your global installation, it may still be inside a sub-directory somewhere as a part of an application that you might have installed using some installer script or the likes.

As stated earlier, monitoring and admin tools for hardware such as RAID controllers are sometimes written in java and may contain log4j. Here are some of the tools that we often use on our hardware:

Adaptec RAID controller

Update Jan. 6 2022: Adaptec has now an official response with patched versions and manual workarounds.

storman, the applicaton that provides the graphical user interface and email alerting for adaptec controllers is java based and uses log4j. Currently Adaptec is working on a new version that uses a newer log4j library. Until then, we recommend to stop the respective service and disable it. However, please note, that you won’t receive email alerts when a disk fails after stopping this utility, so it is important that you regularly check in on your server and manually verify, that your raidsets are still optimal by using the command line interface which is not using log4j.

to disable the manager run:
systemctl stop stor_tomcat
systemctl disable stor_tomcat

to check the raid status with the command line utitliy run:
arcconf list

Areca raid controller

areca utilities are not written in java, you are safe

Intel / LSI Raid Controller

if you are using the newer Intel RAID Web Console 3 (accessible via a web-browser) or LSI Storage Authority, you are not using java, so no Log4j there. The older versions (RWC2 and below) which need to be started as a gui application using a “startui.sh” command, are java based applications but we haven’t analyzed them yet. We recommend to upgrade to RWC3 as it is more user friendly anyway. Visit https://downloadcenter.intel.com to find the latest versions.

Intel Omni-path Fabric Manager GUI

he intel omni-path fabric manager gui uses Log4j. So far we only have found version 1 which is not driectly affected by this vulnerability and should be harmless. However, If you are not using the GUI, uninstall it.

If you find any further tools that you think should be mentioned here of if you find that the information provided on this page is not correct or incomplete, please send us an email to and we will gladly update this page.