Decrypt TLS traffic to Kafka using Wireshark

No Comments

Usually, debugging issues related to TLS in a Java application involves setting the debug flag -Djavax.net.debug=ALL. Unfortunately, the logs will then be flooded with debug entries, which makes troubleshooting the TLS communication difficult. Moreover, sometimes the problem can be solved just by decrypting the TLS traffic.

One way to achieve this is to use jSSLKeyLog, which is a Java Agent Library to log TLS session keys to a file that can be loaded into the Wireshark network protocol analyzer. Coupled with the recently improved Kafka Protocol support in Wireshark, this makes decrypting TLS traffic to/from Kafka captured using the Tcpdump packet sniffer easy.

Procedure

Instrument the Kafka Broker and/or Kafka Client with the jSSLKeyLog agent by adding the JVM flag

-javaagent:/path/to/jSSLKeyLog.jar==/path/to/ssl-key.log

when starting the respective process. The Kafka Broker can be easily instrumented by appending the flag to the KAFKA_OPTS Environment Variable.

Then, use the Tcpdump packet sniffer to capture the traffic to/from the Kafka Broker SSL listener, for example by executing the command

sudo tcpdump -i eth0 -N -A 'port 9093' -w /path/to/trace.pcap

on the machine running the Kafka Broker (adjust the interface, port or path to the packet capture file accordingly).

After the capture finished, open the packet capture file trace.pcap using Wireshark. Without configuring the session key log, the TLS payload will be displayed as “Application Data”.

open tcpdump with Kafka TLS traffic in Wireshark

Next, configure the location of the session key log generated by jSSLKeyLog by choosing Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename in Wireshark.

set session key log in Wireshark

This allows us to inspect the decrypted TLS payload. To also take advantage of the Kafka Decoder in Wireshark, select Analyze > Decode As > Kafka as the dissection protocol.

configure Kafka Decoder in Wireshark

As a result, Wireshark will display the decoded Kafka API Request/Response pairs.

decoded Kafka API Requests and Responses

Demo

The accompanying GitHub repository provides a quick way to test the procedure described above using Docker Compose.

Requirements

The demo uses versions of Wireshark and kafkacat not (yet) available in the standard package repositories.

  • the attached Makefile will build several Docker containers with the needed versions by downloading the source code and compiling the required binaries.
  • the jSSLKeyLog Java Agent will also be built from source and the artifact saved in the tools directory. The directory will then be mounted inside the Docker container running the Kafka Broker and the agent attached via
    KAFKA_OPTS: "-javaagent:/tmp/tools/jSSLKeyLog.jar==/tmp/ssl-key-log/output.log"

To build all the required components, simply run

make build-all

inside the project directory.

Warning: building the required Docker containers requires some time as Wireshark in particular is a complex application.

Running the demo

  • clone the repository
    git clone https://github.com/tudrescu/kafka-wireshark-debug-workshop.git
    cd kafka-wireshark-debug-workshop
  • bootstrap the environment
    make build-all
  • create a local CA, Server and Client certificates
    make certs
  • launch the Docker Compose stack
    docker-compose up -d
  • start a Docker container running tcpdump and attach to the container running the Kafka Broker, see also using tcpdump with Docker.
    docker run \
      --tty \
      --net=container:kafka-1 \
      -v ${PWD}/work/tcpdump-trace:/tmp/tcpdump-trace \
      docker-tcpdump:latest \
      tcpdump -N -A 'port 9093' -w /tmp/tcpdump-trace/trace.pcap
  • enter the kafkacat Docker container and produce some messages
    docker-compose exec kafkacat bash
    kafkacat -b kafka-1:9093 -L -F /tmp/certs/kafkacat.conf
    kafkacat -b kafka-1:9093 -t new_topic -P -F /tmp/certs/kafkacat.conf
  • stop the packet capture using tcpdump
  • open the packet capture using Wireshark. For convenience, the Docker stack provides a containerized Wireshark which is made available in the browser using Xpra. Access the Wireshark via the browser at
    https://localhost:14500/?username=wireshark&password=wireshark
  • configure Wireshark to decode the Kafka TLS traffic as detailed above. The session key log and the packet capture are saved in the ${PWD}\work\ssl-key-log and ${PWD}\work\tcpdump-trace directories, respectively. The directories are mounted in read-only mode inside the Wireshark container under the /home/wireshark/work path.
  • stop and destroy the stack when finished
    docker-compose down -v

Summary

With this approach, we can easily capture and decode TLS traffic to troubleshoot Kafka related issues by inspecting the Kafka protocol requests and responses. The improved support for the dissection of the Kafka protocol available in the latest Wireshark releases might also prove valuable when trying to understand how Kafka works internally.

References

Tudor Udrescu

In a former life, PhD in Applied Mathematics. Worked in IT long enough to see several hype cycles come and go. Currently interested in Kafkalogy.

Comment

Your email address will not be published. Required fields are marked *