How To: Set up TLS Encryption with Kafka

Overview

To secure the connections between Decodable services and your Kafka brokers, you can configure Transport Layer Security (TLS) to encrypt the connection. This ensures that the data between the Decodable services and Kafka brokers is encrypted. In this guide, we'll walk you through how.

You can also configure TLS for client authentication, which is usually known as 2-way authentication or mutual TLS. This will be covered in the next topic.

As a prerequisite, your Kafka brokers need to be configured to accept TLS connections (ref). In the context of Kafka TLS connections, Decodable services are the TLS client and the Kafka brokers are the server.

To use TLS encryption, the client (Decodable services) holds the server's (Kafka brokers) public certificate. Data from the server is encrypted using the server's private key and the client can decrypt it using the public certificate. Data from the client is encrypted using the public certificate and can only be decrypted using the server's private key.

Setup Decodable Kafka Connection

We'll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven't done that yet, refer to the Setup doc.

Create a Stream

decodable stream create --name kafka_tls_in            \
  --description "input stream"                         \
  --field value=string

The stream is used to hold the output of the Kafka source connection in the next step.

Create a Kafka Source TLS Connection

decodable connection create --connector kafka --type source          \
  --name kafka-tls-source                                            \
  --description "Kafka source connection with TLS encryption"        \
  --stream-id=<stream-id>                                            \
  --field value=STRING                                               \
  --prop bootstrap.servers=<broker_list>                  \
  --prop value.format=raw                                            \
  --prop topic=source_raw                                            \
  --prop security.protocol=TLS                                       \
  --prop [email protected]<path_to_cert_file>

To make sure the connection is configured properly:

• broker_list must use the port that accepts TLS connections
• path_to_cert_file must be an X.509 certificate in PEM format
• Your Kafka brokers have a topic source_raw that has data with raw value format

📘

👍Tip: OpenSSL

// If your certificate is in DER format, you can use this command to transform it to PEM format
openssl x509 -in certfile -inform der -outform pem -out cert.pem

Test the Kafka TLS Connection

The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the actual state.

Note that it may take up to 1 minute for the state to update.

Activate the Connection

decodable connection activate <connection_id>

decodable connection get <connection_id>
#sample output
kafka-tls-source
  id                       <connection_id>
  description              Kafka source connection with TLS encryption
  connector                kafka
  type                     source
  stream id                <stream_id>
  schema
    0  body                  STRING
  properties
    bootstrap.servers <broker_list>
    topic                    source_raw
    value.format             raw
    security.protocol        TLS
    tls.broker.certificate   <content_of_cert_file>
  target state             RUNNING
  actual state             RUNNING
  create time              2021-10-13T21:23:33Z
  update time              2021-10-13T21:23:33Z

Create a Preview Job

Run a preview to read from the stream the source Kafka TLS connection writes into. If you produce raw strings to the source_raw topic, you should see sample data coming out from the preview command output.

Note that it may take up to 1 minute for the data to show up.

decodable pipeline preview "SELECT * FROM kafka_tls_in"