To secure the connections between Decodable services and your Kafka brokers, you can configure Transport Layer Security (TLS) to encrypt the connection. This ensures that the data between the Decodable services and Kafka brokers is encrypted. In this guide, we'll walk you through how.
You can also configure TLS for client authentication, which is usually known as 2-way authentication or mutual TLS. This will be covered in the next topic.
As a prerequisite, your Kafka brokers need to be configured to accept TLS connections (ref). In the context of Kafka TLS connections, Decodable services are the TLS client and the Kafka brokers are the server.
To use TLS encryption, the client (Decodable services) holds the server's (Kafka brokers) public certificate. Data from the server is encrypted using the server's private key and the client can decrypt it using the public certificate. Data from the client is encrypted using the public certificate and can only be decrypted using the server's private key.
We'll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven't done that yet, refer to the Setup doc.
decodable stream create --name kafka_tls_in \ --description "input stream" \ --field value=string
The stream is used to hold the output of the Kafka source connection in the next step.
decodable connection create --connector kafka --type source \ --name kafka-tls-source \ --description "Kafka source connection with TLS encryption" \ --stream-id=<stream-id> \ --field value=STRING \ --prop properties.bootstrap.servers=<broker_list> \ --prop value.format=raw \ --prop topic=source_raw \ --prop security.protocol=TLS \ --prop [email protected]<path_to_cert_file>
To make sure the connection is configured properly:
broker_listmust use the port that accepts TLS connections
path_to_cert_filemust be an X.509 certificate in PEM format
- Your Kafka brokers have a topic
source_rawthat has data with
// If your certificate is in DER format, you can use this command to transform it to PEM format
openssl x509 -in certfile -inform der -outform pem -out cert.pem
The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the
Note that it may take up to 1 minute for the state to update.
decodable connection activate <connection_id> decodable connection get <connection_id> #sample output kafka-tls-source id <connection_id> description Kafka source connection with TLS encryption connector kafka type source stream id <stream_id> schema 0 body STRING properties properties.bootstrap.servers <broker_list> topic source_raw value.format raw security.protocol TLS tls.broker.certificate <content_of_cert_file> target state RUNNING actual state RUNNING create time 2021-10-13T21:23:33Z update time 2021-10-13T21:23:33Z
Run a preview to read from the stream the source Kafka TLS connection writes into. If you produce raw strings to the
source_raw topic, you should see sample data coming out from the preview command output.
Note that it may take up to 1 minute for the data to show up.
decodable pipeline preview "SELECT * FROM kafka_tls_in"
Updated 8 months ago