//

Decrypt TLS traffic to Kafka using Wireshark

30.1.2020 | 4 minutes of reading time

Usually, debugging issues related to TLS in a Java application involves setting the debug flag -Djavax.net.debug=ALL. Unfortunately, the logs will then be flooded with debug entries, which makes troubleshooting the TLS communication difficult. Moreover, sometimes the problem can be solved just by decrypting the TLS traffic.

One way to achieve this is to use jSSLKeyLog , which is a Java Agent Library to log TLS session keys to a file that can be loaded into the Wireshark network protocol analyzer. Coupled with the recently improved Kafka Protocol support in Wireshark, this makes decrypting TLS traffic to/from Kafka captured using the Tcpdump packet sniffer easy.

Procedure

Instrument the Kafka Broker and/or Kafka Client with the jSSLKeyLog agent by adding the JVM flag

1-javaagent:/path/to/jSSLKeyLog.jar==/path/to/ssl-key.log
2

when starting the respective process. The Kafka Broker can be easily instrumented by appending the flag to the KAFKA_OPTS Environment Variable.

Then, use the Tcpdump packet sniffer to capture the traffic to/from the Kafka Broker SSL listener, for example by executing the command

1sudo tcpdump -i eth0 -N -A 'port 9093' -w /path/to/trace.pcap
2

on the machine running the Kafka Broker (adjust the interface, port or path to the packet capture file accordingly).

After the capture finished, open the packet capture file trace.pcap using Wireshark. Without configuring the session key log, the TLS payload will be displayed as “Application Data”.

Next, configure the location of the session key log generated by jSSLKeyLog by choosing Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename in Wireshark.

This allows us to inspect the decrypted TLS payload. To also take advantage of the Kafka Decoder in Wireshark, select Analyze > Decode As > Kafka as the dissection protocol.

As a result, Wireshark will display the decoded Kafka API Request/Response pairs.

Demo

The accompanying GitHub repository provides a quick way to test the procedure described above using Docker Compose.

Requirements

The demo uses versions of Wireshark and kafkacat not (yet) available in the standard package repositories.

  • the attached Makefile will build several Docker containers with the needed versions by downloading the source code and compiling the required binaries.
  • the jSSLKeyLog Java Agent will also be built from source and the artifact saved in the tools directory. The directory will then be mounted inside the Docker container running the Kafka Broker and the agent attached via
    1KAFKA_OPTS: "-javaagent:/tmp/tools/jSSLKeyLog.jar==/tmp/ssl-key-log/output.log"
    2

To build all the required components, simply run

1make build-all
2

inside the project directory.

Warning: building the required Docker containers requires some time as Wireshark in particular is a complex application.

Running the demo

  • clone the repository
    1git clone https://github.com/tudrescu/kafka-wireshark-debug-workshop.git
    2cd kafka-wireshark-debug-workshop
    3
  • bootstrap the environment
    1make build-all
    2
  • create a local CA, Server and Client certificates
    1make certs
    2
  • launch the Docker Compose stack
    1docker-compose up -d
    2
  • start a Docker container running tcpdump and attach to the container running the Kafka Broker, see also using tcpdump with Docker .
    1docker run \
    2  --tty \
    3  --net=container:kafka-1 \
    4  -v ${PWD}/work/tcpdump-trace:/tmp/tcpdump-trace \
    5  docker-tcpdump:latest \
    6  tcpdump -N -A 'port 9093' -w /tmp/tcpdump-trace/trace.pcap
    7
  • enter the kafkacat Docker container and produce some messages
    1docker-compose exec kafkacat bash
    2kafkacat -b kafka-1:9093 -L -F /tmp/certs/kafkacat.conf
    3kafkacat -b kafka-1:9093 -t new_topic -P -F /tmp/certs/kafkacat.conf
    4
  • stop the packet capture using tcpdump
  • open the packet capture using Wireshark. For convenience, the Docker stack provides a containerized Wireshark which is made available in the browser using Xpra . Access the Wireshark via the browser at
    1https://localhost:14500/?username=wireshark&password=wireshark
    2
  • configure Wireshark to decode the Kafka TLS traffic as detailed above . The session key log and the packet capture are saved in the ${PWD}\work\ssl-key-log and ${PWD}\work\tcpdump-trace directories, respectively. The directories are mounted in read-only mode inside the Wireshark container under the /home/wireshark/work path.
  • stop and destroy the stack when finished
    1docker-compose down -v
    2

Summary

With this approach, we can easily capture and decode TLS traffic to troubleshoot Kafka related issues by inspecting the Kafka protocol requests and responses. The improved support for the dissection of the Kafka protocol available in the latest Wireshark releases might also prove valuable when trying to understand how Kafka works internally.

References

share post

Likes

0

//

More articles in this subject area\n

Discover exciting further topics and let the codecentric world inspire you.

//

Gemeinsam bessere Projekte umsetzen

Wir helfen Deinem Unternehmen

Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.

Hilf uns, noch besser zu werden.

Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.