Documentation
Community SlackApplication

How To: Set up SASL Authentication with Kafka

Suggest Edits

Overview

A common method for securing client connections to Kafka brokers is through SASL (ref)

Decodable supports a number of SASL authentication mechanisms, both with and without SSL/TLS encryption.

Setup Decodable Kafka Connection

We'll assume here that you already have a Decodable account and have gotten started with the Decodable CLI. If you haven't done that yet, refer to the Setup doc.

Create a Stream

decodable stream create --name kafka_sasl_in            \
  --description "input stream"                          \
  --field value=string

Create a Connection

When configuring a SASL connection, the required properties are:

security-protocol (one of SASL_SSL or SASL_PLAINTEXT)
sasl.mechanism (one of PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512)
sasl.username (the provided username or API key)
sasl.password (the provided password or API secret key)

🚧

Self-Signed Certificates

Note that when using SASL authentication with SSL/TLS encryption (SASL_SSL), the broker must be configured to use a trusted certificate. Self-signed certificates are not supported at this time.

These can be used as Decodable connection properties, e.g.

decodable connection create --connector kafka --type source          \
  --name kafka-sasl-source                                           \
  --description "Kafka source connection with SASL/SSL"              \
  --stream-id=<stream-id>                                            \
  --field value=STRING                                               \
  --prop value.format=raw                                            \
  --prop bootstrap.servers=<broker_list>                             \
  --prop topic=source_raw                                            \
  --prop security.protocol=SASL_SSL                                  \
  --prop sasl.mechanism=PLAIN                                        \
  --prop sasl.username="<username>"                                  \
  --prop sasl.password="<password>"

Test the Connection

The quickest way to test the connection is to activate it and run a preview job. After activation, we can verify that the connection is activated successfully by checking the actual state.

Note that it may take up to 1 minute for the state to update.

Activate the Connection

decodable connection activate <connection_id>

decodable connection get <connection_id>
#sample output
kafka-sasl-source
  id                       a3ead34d
  description              
  connector                kafka
  type                     source
  stream id                abbcfccd
  schema
    0  value                 STRING
  properties
    bootstrap.servers        <broker_list>
    value.format             raw
    sasl.mechanism           PLAIN
    sasl.username            <username>
    security.protocol        SASL_SSL
    topic                    source_raw
  target state             RUNNING
  actual state             RUNNING
  create time              2021-11-23T15:15:02Z
  update time              2021-11-23T15:15:02Z

Create a Preview Job

Run a preview to read from the stream the source Kafka connection writes into. If you produce raw strings to the source_raw topic, you should see sample data coming out from the preview command output.

Note that it may take up to 1 minute for the data to show up.

decodable pipeline preview "SELECT * FROM kafka_sasl_in"

Updated over 1 year ago