Overview

Analysing malicious Java applets at runtime using the JVMTI

No Comments

When your browser asks you if you want to allow Java applets for a certain website, you might be interested about what that applet actually does when you execute it on your machine and whether or not it is safe to do so. What you can do is to use recent anti-virus solutions but they usually don’t provide you with too much details. Thus, a more interesting approach is to create a little tool that outlines the functionality of an applet at a reasonable level of abstraction, so you can easily get a grasp on what is going on under the hood.

In this article, I’d like to give you a brief insight into analysing Java applets using the Java virtual machine tool interface (JVMTI) [1]. On this occasion, I will also take a litte detour through the dark side of Java applet writing and Java byte code obfuscation. Although I’m going to focus on analysing malicious Java applets, the presented approach can of course be applied to benign applets or other Java applications as well.

The approach I’m going to present here can be loosely described as an automated debugging process. The overall goal is to analyse the behaviour of an examined applet at runtime by observing performed Java API calls in order to detect sandbox breach attempts. The reason why I’m doing all this is that it is really hard to determine the functionality of a malicious Java applet by just looking at the decompiled code. As you can see in the following example (taken from the infamous Blackhole Exploit Kit), the entire code is highly obfuscated and thus effectively disguising its evil purpose. One technique commonly used for obfuscation is to replace method calls or class names with reflections and decrypt used identifiers at runtime. This can also be taken to the next level by encrypting entire classes and decrypting them before loading at runtime.

ClassLoader var2 = Addo.class.getClassLoader();
Class var3 = var2.loadClass(pimyw("SXOX\027W\\M\027lku"));
Class[] var69 = { String.class };
Constructor c = var3.getConstructor(var69);
Object[] var70 = { urls[kk] };
Object var4 = c.newInstance(var70);
String var73 = pimyw("VI\\WjMK\\XT");
is = (InputStream)dIaAdXI(null, var4, var73, new Object[0]);
String ss = ByteArrayOutputStream.class.getName();
Object[] var51 = new Object[0];
Class[] var56 = new Class[0];
fos = (ByteArrayOutputStream)TRwRxYhJ(ss, var51, var56);

To automatically analyse the applet at runtime, you can compile an agent library using the JVMTI, providing virtually unlimited access to the JVM. Therefore, the agent first needs to inform the JVM about the events it is actually interested in and register respective callbacks. This can be done by implementing the Agent_OnLoad method that is called automatically by the JVM when the agent library is loaded. The following listing shows a simple example obtaining the JVMTI environment object that is used for further initialisations.

JNIEXPORT jint JNICALL
Agent_OnLoad(JavaVM *vm, char *options, void *reserved) {
  jvmtiEnv* env;
  assert(JVMTI_ERROR_NONE == vm->GetEnv(reinterpret_cast<void**>(&env), JVMTI_VERSION));
 
  RegisterCallbacks(env);
  return JNI_OK;
}

Since we are only interested on interactions of the Java applet with the Java API, we first identify all API methods called by the applet including calls using the reflection API. Therefore, the agent can use the CLASS_FILE_LOAD_HOOK event to analyse the byte code of every class being loaded. This not only allows for determining method calls but also to detect, and in some cases to even repair, invalid byte code sequences that could be used to attack the JVM. Luckily, this event is also generated for classes dynamically generated at runtime, so we will not miss anything that is being loaded. In the following example, the two callbacks ClassPrepareCallback and BreakpointCallback are registered and enabled, so they will be called by the JVM whenever the respective event occurs.

void
RegisterCallbacks(jvmtiEnv* env) {
  /* Ensure that the JVM provides the required capabilities */
  jvmtiCapabilities capabilities = { 0 };
  capabilities.can_generate_breakpoint_events = 1;
  assert(JVMTI_ERROR_NONE == env->AddCapabilities(&capabilities));
 
  /* Register callbacks */
  jvmtiEventCallbacks callbacks = { 0 };
  callbacks.ClassPrepare = &ClassPrepareCallback;
  callbacks.Breakpoint = &BreakpointCallback;
  assert(JVMTI_ERROR_NONE == env->SetEventCallbacks(&callbacks, sizeof(callbacks)));
 
  /* Enable events */
  assert(JVMTI_ERROR_NONE == env->SetEventNotificationMode(JVMTI_ENABLE, 
         JVMTI_EVENT_BREAKPOINT, NULL));
  assert(JVMTI_ERROR_NONE == env->SetEventNotificationMode(JVMTI_ENABLE, 
         JVMTI_EVENT_CLASS_PREPARE, NULL));
}

With all method calls extracted from loaded class files, we can use the JVMTI to automatically apply breakpoints. Therefore, we use a callback for the CLASS_PREPARE event, which is triggered when a class has been successfully loaded by the JVM and is ready to be used, but before any code has been executed. If our callback is called, we can use GetClassMethods(jclass klass, jint* method_count_ptr, jmethodID** methods_ptr) to find the methods we are interested in and use SetBreakpoint(jmethodID method, jlocation location) to set the breakpoints. Since we want to break at the very first instruction of every method, we use 0 for the location.

With all breakpoints in place, we can use our callback method for the BREAKPOINT event to intercept respective calls. Whenever one of these methods is called, the JVM hands over the flow control to the agent library. The following example shows the breakpoint callback’s signature and outlines briefly how it can be used to access provided parameters. The actual code is pretty much straight forward, but it requires some string parsing and ugly switch statements, so I left it out for the sake of brevity. Using this callback, the agent is able to inspect the current call stack to reconstruct all parameters that were passed in to the method.

void JNICALL
BreakpointCallback(jvmtiEnv *jvmti_env,
JNIEnv* jni_env,
jthread thread,
jmethodID method,
jlocation location) {
 
/* Get method name, signature and arguments using:
* jvmti_env->GetMethodName for method name and signature
*
* Parse signature to get parameter types and order
* jvmti_env->GetLocalObject to access parameters
*/
}

Additionally, the JVMTI provides further functionality to modify passed in parameters or even to immediately return and thus to prevent the method from being executed at all. This can be used by the agent for example to prevent the applet from tampering with the local file system or to provide it with fake information about its runtime environment. It is, however, necessary to check the stack trace for the calling class first, so that respective methods can function as expected when called by Java core classes.

Another neat feature of the JVMTI is that it provides access to the JNI [2], allowing for even more interactions with the examined applet. One particularly interesting feature of the JNI is the capability to transform reflected method invocations back to the original method by calling jni_env->FromReflectedMethod(jobject method). We can use this method to apply breakpoints to the reflected methods right before they are being called. Thus, by applying breakpoints to the reflection API and adding breakpoints to the resolved methods, our agent is able to process respective calls just like any other method call.

As a result, we are able to automatically create high level behaviour traces of an examined applet as you can see in the example below. What you see is the execution trace of the obfuscated code shown earlier in this post as well as a few following instructions. The most interesting thing you can see here is the second last line, showing the applet writing binary data to a file. As you can see at the first two bytes, the file being written is a MS DOS executable, which is intended to perform the actual malware infection of the executing host. Since we only track interactions of the applet with the Java API, the resulting trace is agnostic against encrypted identifiers or class code and is not cluttered by repeated calls (e.g. for decrypting identifiers) within the applet.

Addo -> java.lang.ClassLoader.loadClass("java.io.InputStream");
Addo -> java.lang.ClassLoader.loadClass("java.io.ByteArrayOutputStream");
Addo -> java.lang.Class.getName();
Addo -> java.lang.reflect.Constructor.newInstance([ ]); // java.io.ByteArrayOutputStream.()V
Addo -> java.lang.ClassLoader.loadClass("java.lang.Runtime");
Addo -> java.lang.Runtime.getRuntime();
Addo -> java.io.ByteArrayOutputStream.write([ '<', 'h', 't', 'm', 'l', '>' ... );
Addo -> java.io.ByteArrayOutputStream.close();
Addo -> java.lang.ClassLoader.loadClass("java.io.FileOutputStream");
Addo -> java.lang.Class.getName();
Addo -> java.lang.reflect.Constructor.newInstance([ "C:\\Users\\agent\\AppData\\Local\\Tempuhhkaovuebenitwiwiw.bfg" ]); // java.io.FileOutputStream.(Ljava/lang/String;)V
sun.reflect.NativeConstructorAccessorImpl -> java.io.FileOutputStream.("C:\\Users\\agent\\AppData\\Local\\Tempuhhkaovuebenitwiwiw.bfg");
Addo -> java.io.FileOutputStream.write([ 'M', 'Z', 0x90, 0x00, '_', 0xb0, 0xe4, ... ]);
Addo -> java.io.FileOutputStream.close();

The information from the behaviour trace can be used to determine malicious behaviour by detecting methods used to break out of the Java applet sandbox. Malicious applets usually exploit security holes to execute a privileged method call allowing them for example to disable Java’s default security manager. This, however, requires the applet to call System.setSecurityManager(null) which can unambiguously be considered malicious behaviour. Other examples are tampering with the file system, loading native libraries or creating a custom class loader. Using respective API calls as an heuristic, you can identify malicious applets even without observing the actual exploit.

At the end of September, I presented a more extensive version of the described approach at the 9th annual international conference of malicious and unwanted software. It was a rather intimate conference with the major goal of bringing together research and industry and to encourage controversial discussions. The keynote of Giovanni Vigna already showed that this was not merely a theory, as it started a heated discussion about static and dynamic analysis techniques no later than on slide two. Overall I pretty much enjoyed the conference and came back with a lot of new insights and ideas.

I hope I was able to provide you with an interesting introduction in Java applet analysis using the JVMTI and detecting malicious activities in the behaviour observed. Feel free to contact me directly if you have any further questions or leave a comment right below this post.

References

[1] http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
[2] http://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html

Comment

Your email address will not be published. Required fields are marked *