How To: Set up TLS encryption with Apache Kafka

To secure the connections between Decodable services and your Apache Kafka brokers, you can configure Transport Layer Security (TLS) to encrypt the connection. This ensures that the data between the Decodable services and Kafka brokers is encrypted. In this guide, we’ll walk you through how.

Consult the Apache Kafka source connector page for full details and configuration reference.

You can also configure TLS for client authentication, which is usually known as 2-way authentication or mutual TLS. This is covered here.

As a prerequisite, your Kafka brokers need to be configured to accept TLS connections (ref). In the context of Kafka TLS connections, Decodable services are the TLS client and the Kafka brokers are the server.

To use TLS encryption, the client (Decodable services) holds the server’s (Kafka brokers) public certificate. Data from the server is encrypted using the server’s private key and the client can decrypt it using the public certificate. Data from the client is encrypted using the public certificate and can only be decrypted using the server’s private key.

Setup Decodable Kafka connection

We’ll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven’t done that yet, see The Decodable CLI to learn how to install and setup the Decodable CLI.

Create a Stream

decodable stream create --name kafka_tls_in            \
  --description "input stream"                         \
  --field value=string

The stream is used to hold the output of the Kafka source connection in the next step.

Create a Kafka source TLS connection

Here is an example resource YAML that describes a Kafka TLS connection you can create using the Decodable CLI:

---
kind: connection
metadata:
  name: kafka-tls-source
  description: Kafka source connection with TLS encryption
spec_version: v2
spec:
  connector: kafka
  type: source
  stream_mappings:
    - stream_name: kafka_tls_in
      external_resource_specifier:
        topic: source_topic
  properties:
    value.format: json
    bootstrap.servers: <broker_list>
    security.protocol: TLS
    tls.broker.certificate: <cert>

To make sure the connection is configured correctly:

broker_list must use the port that accepts TLS connections
cert must be an X.509 certificate in PEM format
Your Kafka brokers have a topic source_topic that has data with json value format

If your certificate is in DER format, you can use this command to transform it to PEM format: openssl x509 -in certfile -inform der -outform pem -out cert.pem

Test the Kafka TLS connection

The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the actual state.

Note that it can take up to 1 minute for the state to update.

Activate the connection

$ decodable connection activate <connection_id>

$ decodable connection get <connection_id>

kafka-tls-source
id                          <connection_id>
description
connector                   kafka
type                        source
stream mappings
  stream id                 <stream_id>
  external resource specifier
    topic                   source_topic
schema
properties
  value.format:             json
  bootstrap.servers:        <broker_list>
  security.protocol:        TLS
  tls.broker.certificate   <cert>

Create a preview job

Run a preview to read from the stream the source Kafka TLS connection writes into. If you produce JSON strings to the source_topic topic, you should see sample data coming out from the preview command output.

Note that it may take up to 1 minute for the data to appear.

decodable pipeline preview "SELECT * FROM kafka_tls_in"