How To: Set up SASL Authentication with Kafka
Overview
A common method for securing client connections to Kafka brokers is through SASL (ref)
Decodable supports a number of SASL authentication mechanisms, both with and without SSL/TLS encryption.
Setup Decodable Kafka Connection
We'll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven't done that yet, refer to the Setup doc.
Create a Stream
decodable stream create --name kafka_sasl_in \
--description "input stream" \
--field value=string
Create a Connection
When configuring a SASL connection, the required properties are:
security-protocol
(one of SASL_SSL
or SASL_PLAINTEXT
)
sasl.mechanism
(one of PLAIN
, SCRAM-SHA-256
or SCRAM-SHA-512
)
sasl.username
(the provided username or API key)
sasl.password
(the provided password or API secret key)
Self-Signed Certificates
Note that when using SASL authentication with SSL/TLS encryption (
SASL_SSL
), the broker must be configured to use a trusted certificate. Self-signed certificates are not supported at this time.
These can be used as Decodable connection properties, e.g.
decodable connection create --connector kafka --type source \
--name kafka-sasl-source \
--description "Kafka source connection with SASL/SSL" \
--stream-id=<stream-id> \
--field value=STRING \
--prop value.format=raw \
--prop bootstrap.servers=<broker_list> \
--prop topic=source_raw \
--prop security.protocol=SASL_SSL \
--prop sasl.mechanism=PLAIN \
--prop sasl.username="<username>" \
--prop sasl.password="<password>"
Test the Connection
The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the actual state
.
Note that it may take up to 1 minute for the state to update.
Activate the Connection
decodable connection activate <connection_id>
decodable connection get <connection_id>
#sample output
kafka-sasl-source
id a3ead34d
description
connector kafka
type source
stream id abbcfccd
schema
0 value STRING
properties
bootstrap.servers <broker_list>
value.format raw
sasl.mechanism PLAIN
sasl.username <username>
security.protocol SASL_SSL
topic source_raw
target state RUNNING
actual state RUNNING
create time 2021-11-23T15:15:02Z
update time 2021-11-23T15:15:02Z
Create a Preview Job
Run a preview to read from the stream the source Kafka connection writes into. If you produce raw strings to the source_raw
topic, you should see sample data coming out from the preview command output.
Note that it may take up to 1 minute for the data to show up.
decodable pipeline preview "SELECT * FROM kafka_sasl_in"
Updated over 1 year ago