Analysing malicious Java applets at runtime using the JVMTI

No Comments

When your browser asks you if you want to allow Java applets for a certain website, you might be interested about what that applet actually does when you execute it on your machine and whether or not it is safe to do so. What you can do is to use recent anti-virus solutions but they usually don’t provide you with too much details. Thus, a more interesting approach is to create a little tool that outlines the functionality of an applet at a reasonable level of abstraction, so you can easily get a grasp on what is going on under the hood.

In this article, I’d like to give you a brief insight into analysing Java applets using the Java virtual machine tool interface (JVMTI) [1]. On this occasion, I will also take a litte detour through the dark side of Java applet writing and Java byte code obfuscation. Although I’m going to focus on analysing malicious Java applets, the presented approach can of course be applied to benign applets or other Java applications as well.

The approach I’m going to present here can be loosely described as an automated debugging process. The overall goal is to analyse the behaviour of an examined applet at runtime by observing performed Java API calls in order to detect sandbox breach attempts. The reason why I’m doing all this is that it is really hard to determine the functionality of a malicious Java applet by just looking at the decompiled code. As you can see in the following example (taken from the infamous Blackhole Exploit Kit), the entire code is highly obfuscated and thus effectively disguising its evil purpose. One technique commonly used for obfuscation is to replace method calls or class names with reflections and decrypt used identifiers at runtime. This can also be taken to the next level by encrypting entire classes and decrypting them before loading at runtime.

ClassLoader var2 = Addo.class.getClassLoader();
Class var3 = var2.loadClass(pimyw("SXOX\027W\\M\027lku"));
Class[] var69 = { String.class };
Constructor c = var3.getConstructor(var69);
Object[] var70 = { urls[kk] };
Object var4 = c.newInstance(var70);
String var73 = pimyw("VI\\WjMK\\XT");
is = (InputStream)dIaAdXI(null, var4, var73, new Object[0]);
String ss = ByteArrayOutputStream.class.getName();
Object[] var51 = new Object[0];
Class[] var56 = new Class[0];
fos = (ByteArrayOutputStream)TRwRxYhJ(ss, var51, var56);

To automatically analyse the applet at runtime, you can compile an agent library using the JVMTI, providing virtually unlimited access to the JVM. Therefore, the agent first needs to inform the JVM about the events it is actually interested in and register respective callbacks. This can be done by implementing the Agent_OnLoad method that is called automatically by the JVM when the agent library is loaded. The following listing shows a simple example obtaining the JVMTI environment object that is used for further initialisations.

Agent_OnLoad(JavaVM *vm, char *options, void *reserved) {
  jvmtiEnv* env;
  assert(JVMTI_ERROR_NONE == vm->GetEnv(reinterpret_cast<void**>(&env), JVMTI_VERSION));
  return JNI_OK;

Since we are only interested on interactions of the Java applet with the Java API, we first identify all API methods called by the applet including calls using the reflection API. Therefore, the agent can use the CLASS_FILE_LOAD_HOOK event to analyse the byte code of every class being loaded. This not only allows for determining method calls but also to detect, and in some cases to even repair, invalid byte code sequences that could be used to attack the JVM. Luckily, this event is also generated for classes dynamically generated at runtime, so we will not miss anything that is being loaded. In the following example, the two callbacks ClassPrepareCallback and BreakpointCallback are registered and enabled, so they will be called by the JVM whenever the respective event occurs.


Your email address will not be published. Required fields are marked *