corretto/hotpatch-for-apache-log4j2

An agent to hotpatch the log4j RCE from CVE-2021-44228.

Log4jHotPatch

This is a tool which injects a Java agent into a running JVM process. The agent will attempt to patch the lookup() method of all loaded org.apache.logging.log4j.core.lookup.JndiLookup instances to unconditionally return the string "Patched JndiLookup::lookup()". It is designed to address the CVE-2021-44228 remote code execution vulnerability in Log4j without restarting the Java process. This tool will also address CVE-2021-45046.

This has been currently only tested with JDK 8, 11, 15 and 17 on Linux!

Building

Gradle

To build on linux, mac and Windows subsystem for linux

./gradlew build

To build on Windows

.\gradlew.bat build

Depending on the platform you are building. This will generate build/libs/Log4jHotPatch.jar

Maven

To build using Maven use

mvn clean package

This will generate a target/Log4jHotPatch.jar.

Running

JDK 8

java -cp <java-home>/lib/tools.jar:Log4jHotPatch.jar Log4jHotPatch <java-pid>

JDK 11 and newer

java -jar Log4jHotPatch.jar <java-pid>

Running the static agent

Simply add the agent to your java command line as follows:

java -classpath <class-path> -javaagent:Log4jHotPatch.jar <main-class> <arguments>

Testing the agent

There are a set of tests that can be run outside Gradle or Maven.

build-tools/bin/run_tests.sh Log4jHotPatch.jar <JDK_ROOT>

Known issues

If you get an error like:

Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: The VM does not support the attach mechanism
	at jdk.attach/sun.tools.attach.HotSpotAttachProvider.testAttachable(HotSpotAttachProvider.java:153)
	at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:56)
	at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
	at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:115)
	at Log4jHotPatch.main(Log4jHotPatch.java:139)

this means that your JVM is refusing any kind of help because it is running with -XX:+DisableAttachMechanism.

If you get an error like:

com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
	at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
	at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:182)
	at Log4jHotPatch.main(Log4jHotPatch.java:259)

this means you're running as a different user (including root) than the target JVM. JDK 8 can't handle patching as root user (and triggers a thread dump in the target JVM which is harmless). In JDK 11 patching a non-root process from a root process works just fine.

If you get an error like this in the target process:

Exception in thread "Attach Listener" java.lang.ExceptionInInitializerError
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
Caused by: java.security.AccessControlException: access denied ("java.util.PropertyPermission" "log4jFixerAgentVersion" "write")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.security.AccessController.checkPermission(AccessController.java:886)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at java.lang.System.setProperty(System.java:794)
        at Log4jHotPatch.<clinit>(Log4jHotPatch.java:66)

it means the target process has a security manager installed. Look for this command line option in the target process:

-Djava.security.policy=/local/apollo/.../apollo-security.policy

If you encounter this error, make sure you are using the latest version of the tool

Important: If you attempted to patch as the wrong user, you may need to delete .attach_pid<pid> files (found in /tmp and/or the CWD of the VM process) before trying again. These files need to have the right ownership for attach to succeed.

hotpatch-for-apache-log4j2's Issues

Migrate testing script to gradle

The current tests are based on a simple shell script. Move those tests into gradle and stop including a log4j jar for testing.

/tmp mount points in hot patch

The hot patch created a lot of /tmp mount points. Since my customer collects metrics on mount points, these /tmp mount points cost customer a lot on these unwanted metrics. Could these mount points be disabled ?

JDK 8 - Error: Could not find or load main class <PID>

When trying to run

java -cp <java-home>/lib/tools.jar:Log4jHotPatch.jar <java-pid>

with JDK 8, I only get the message

Error: Could not find or load main class <PID>

what could i be doing wrong?

java.io.FileNotFoundException: /tmp/agentXXXXXX.jar issue at tomcat startup avoiding the webapp to start

Hi,
The patch has been deployed in AL1 on one of our server via "yum update --security", and we had a problem with it.
We have the following error in the tomcat log avoiding any web applications to start on this server :

java.io.FileNotFoundException: /tmp/agent693101429317028124.jar (Aucun fichier ou dossier de ce type)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:228)
at java.util.zip.ZipFile.(ZipFile.java:157)
at java.util.jar.JarFile.(JarFile.java:169)
at java.util.jar.JarFile.(JarFile.java:106)
at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
at org.apache.tomcat.util.scan.FileUrlJar.(FileUrlJar.java:48)
at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:34)
at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1957)
at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1932)
at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1917)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1322)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5566)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1017)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:993)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

In fact, this server has a specifiticy : there are 20 concurrent JVM / Tomcat instances running and launched via chkconfig services... I'm not sure but this could be the root cause of the pb...
We applied the "sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill" to deactivate the HotPatch, and it worked.
So it confirms that the issue is due to the Log4j HotPatch.
I think you should publish a fix for this because I assume we won't be the only impacted customers.

A precision : we use amazon-corretto-8.302.08.1-linux-x64 under AL1

Best Regards.

Add CHANGELOG.md

We need to include a proper CHANGELOG.md with all the changes we have added since the initial release.

Unable to install the patch in MacOs

Environment:

macOs Monterey 12.1
java -version
openjdk version "11.0.9.1" 2020-11-04 LTS
OpenJDK Runtime Environment Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS, mixed mode)

I tried to run the below command :
jar -cfm Log4jHotPatch.jar Manifest.mf *.class

Got the error below:

java.io.FileNotFoundException: Manifest.mf (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at jdk.jartool/sun.tools.jar.Main.run(Main.java:267)
at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)

Am i missing something.

More generic?

I'm curious if you can make something more generic with gdb that will mitigate future exploits that leak environment variables?

https://stackoverflow.com/questions/205064/is-there-a-way-to-change-the-environment-variables-of-another-process-in-unix

Also for that matter, use something at the Linux level to sanitize a process's network connection from doing the JDNI lookups?

Hotpatch installation reports error Failed to get D-Bus connection: Operation not permitted

Steps to reproduce :

If you try to install any one of these procps-ng/net-tools/iputils before installing corretto, then log4j hotpatch will report following error while installation, however installation completed.
Dockerfile :

FROM amazonlinux:2 
RUN yum update -y
RUN yum install -y procps-ng net-tools iputils
RUN amazon-linux-extras enable corretto8
RUN yum install -y java-1.8.0-amazon-corretto-devel

$ docker build -t log4j .

Installation of log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch reports following error :

Failed to get D-Bus connection: Operation not permitted
Created symlink /etc/systemd/system/multi-user.target.wants/log4j-cve-2021-44228-hotpatch.service, pointing to /usr/lib/systemd/system/log4j-cve-2021-44228-hotpatch.service.
Failed to get D-Bus connection: Operation not permitted
warning: %post(log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch

As a workaround, you can try installing procps-ng/net-tools/iputils these after corretto.

Q: Does this work with shaded log4j2?

If I understand the code correctly, package name is included in checking the impacted class. As shading would change the package name, this implies that the solution doesn't work with uber jars where log4j has been shaded to avoid version conflicts.

Can you confirm this?

Cannot attach `Log4jHotPatch` agent to official releases of Jenkins

Note: All instructions here assume Java 11 (OpenJDK), but I was also able to reproduce the issue with Java 8 (OpenJDK).

Steps to reproduce

• Download an official release of the Jenkins generic Java package (.war).
• Start Jenkins with java -jar jenkins.war.
• Build Log4jHotPatch and run java -Dlog4jFixerVerbose=true -cp Log4jHotPatch.jar Log4jHotPatch ${JENKINS_PID}.

Expected results

Note: These are the actual results with a local (non-official) build of Jenkins:

The Log4jHotPatch agent is attached successfully.

Actual results

The Log4jHotPatch agent cannot be attached:

com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:165)
	at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:228)
	at Log4jHotPatch.main(Log4jHotPatch.java:301)
Error: couldn't loaded the agent into JVM process 546677

The Jenkins logs show:

Exception in thread "Attach Listener" java.lang.SecurityException: class "Log4jHotPatch"'s signer information does not match signer information of other classes in the same package
	at java.base/java.lang.ClassLoader.checkCerts(ClassLoader.java:1151)
	at java.base/java.lang.ClassLoader.preDefineClass(ClassLoader.java:906)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1015)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:431)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Agent failed to start!

Evaluation

Stepping through the problematic frames in the debugger, I do not think Jenkins is at fault here. Jenkins contains some classes in the unnamed package namespace:

$ jar tf jenkins.war
[…]
ColorFormatter.class
JNLPMain.class
LogFileOutputStream$1.class
LogFileOutputStream.class
Main$FileAndDescription.class
Main.class
MainDialog.class
[…]
META-INF/JENKINS.RSA
META-INF/JENKINS.SF
META-INF/MANIFEST.MF
[…]
$

Note that META-INF/JENKINS.RSA and META-INF/JENKINS.SF are only present in our signed official releases; a local build will not contain these. The presence of these classes and signatures means that the system class loader associates the signatures with the unnamed package namespace. Then, when Log4jHotPatch attempts to load a class into the same unnamed package namespace without a signature, class loading fails.

I do not think Log4jHotPatch should make any assumptions about whether or not it is free to load unsigned classes into the unnamed package namespace. Perhaps Log4jHotPatch should use its own package namespace instead.

Publish binary artifacts

Build and publish the current jar output and validate it can be used with at least 8, 11, 15 and 17

Apache Oozie job failing with the hotpatch

Apache Oozie job intermittently fails after applying the hotpatch. This is because Oozie will list and the print the file and directory name in the current directory. On the other hand, the hotpatch creates attach_pid file in the current directory to attach the JVM and then delete it. When the attach_pid file is deleted between the listing and the printing, the Oozie job fails by NoSuchFileException. The below is the stacktrace:

java.nio.file.NoSuchFileException: ./.attach_pid1517
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
    at java.nio.file.Files.walkFileTree(Files.java:2706)
    at org.apache.oozie.action.hadoop.LocalFsOperations.printContentsOfDir(LocalFsOperations.java:59)
    at org.apache.oozie.action.hadoop.LauncherAM.printDebugInfo(LauncherAM.java:293)
    at org.apache.oozie.action.hadoop.LauncherAM.access$100(LauncherAM.java:55)
    at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:221)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

The mitigation is to disable the hotpatch on the Oozie server.

I plan to provide a fix to Oozie to retry the directory listing, however, it is worth creating the issue here for searchability. Thanks.

We are seeing logging to to stdout which affects parsing of our outputs

We are seeing logging to to stdout which affects parsing of our outputs.

I have tried:

   <sysproperty key="log4jFixerVerbose" value="false" escape="false"/>

However, we are still getting the log messages like:

Loading Java Agent version 1 (using ASM6).
Patching class org.apache.logging.log4j.core.lookup.JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)

Any idea why the sysproperty might be ignored? Thanks!

Setup automated testing and integration

Use and improve the current tests and setup github actions for them

Error: couldn't loaded the agent into JVM process

I seen this issue, I'm using java8

com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:121) at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:234) at Log4jHotPatch.main(Log4jHotPatch.java:298) at Log4jHotPatch17.main(Log4jHotPatch17.java:18) Error: couldn't loaded the agent into JVM process 1234 Are you running as a different user (including root) than process 1234? Errors occurred deploying hot patch. If you are using java 8 to run this tool against JVM 11 or later, the target JVM may still be patched. Please look for a message like 'Loading Java Agent (using ASM 6).' in stdout of the target JVM. Also note that JVM 17+ are not supported.

unicode characters causing javac errors

Thank you for posting this patch to the public. I fixed the issue locally by simply deleting the unicode characters, would still like to report it. The issue occurred in the License section of comments.

javac -XDignore.symbol.file=true -cp %JAVA_HOME%/lib/tools.jar Log4jHotPatch.java
Log4jHotPatch.java:4: error: unmappable character for encoding Cp1252

• Licensed under the Apache License, Version 2.0 (the ΓÇ£LicenseΓÇ?).
  ^
  Log4jHotPatch.java:10: error: unmappable character for encoding Cp1252
• or in the ΓÇ£licenseΓÇ? file accompanying this file. This file is distributed
  ^
  Log4jHotPatch.java:11: error: unmappable character for encoding Cp1252
• on an ΓÇ£AS ISΓÇ? BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
  ^
  3 errors

Add process information to log when a java process is detected

Amazon Linux 1 instances running a variety of Java versions.

Our security log watcher was detecting a user logging in which is a service account and should not log in. The timestamps on these events are within 1-3 seconds of the log4jhotpach firing and logging to /var/log/messages

However, it was impossible for us to see what the process was as we have several short-lived processes which are gone by the time we receive the alert.

I modified the log4j-cve-2021-44228-hotpatch script to log more information so we can better identify they detected process. I added this line of code

log "PID Info $(ps -o user:12,pid,ppid,pgid,sid,stat,time,lstart,cmd f ${pid})"

at line 114 of the 1.1.12 version of the patch

This helped us find the misbehaving processes.

Please consider adding something like this to the hotpatch

No protection from combined attack with CVE-2021-45105

With CVE-2021-45105 a malicious user can cause a DoS which in most scenarios will lead to JVM restart.

After restart there's a time window when an attack against CVE-2021-45046 or CVE-2021-44228 could be launched before the agent has attached to the JVM process.

Would it be possible to extend this solution to cover this scenario as well?

error installing in ubuntu 20.04

The command gives me the following error:

# ./gradlew build --warning-mode all

> Configure project :
The AbstractArchiveTask.version property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveVersion property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:version for more details.
        at build_34dfj8owb22lztkrt6bemzzq5$_run_closure2.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:17)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
The AbstractArchiveTask.archiveName property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveFileName property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:archiveName for more details.
        at build_34dfj8owb22lztkrt6bemzzq5$_run_closure3.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:34)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

FAILURE: Build failed with an exception.

* Where:
Build file '/root/hotpatch-for-apache-log4j2/build.gradle' line: 46

* What went wrong:
A problem occurred evaluating root project 'hotpatch-for-apache-log4j2'.
> java.lang.NullPointerException (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s

Should I add debug output as well?

Create build system for distro maintainers

Using recent Gradle or Maven with plugins on some older distro builds systems is problematic.

Create a parallel system that builds the same artifacts
Import ASM source and update license, 3p attributions
Shade package name and imports in source
Update gradle to use source build of ASM