Use the Apache Kafka Connector to send data to Apache Kafka. If you are looking for information about how to create a connection to get data into Decodable from Kafka, see Apache Kafka in the Connect to a data source chapter.

Overview

Additional Kafka Support
If your version of Kafka is not currently supported, please contact [email protected] or join our Slack community and let us know!

Create a connection to Apache Kafka with the Apache Kafka Connector

Follow these steps to get data from or send data to Apache Kafka. These steps assume that you are using the Decodable web interface. However, if you want to use the Decodable CLI to create the connection, you can refer to the Property Name column for information about what the underlying property names are.

1. From the Connections page, select the Apache Kafka Streaming Connector and complete the following fields.

2. Select which stream contains the records that you’d like to send to Apache Kafka. Then, select Next.
3. Give the newly created connection a Name and Description and select Save.

Reference

Security

Decodable supports four different security protocols for accessing Kafka brokers.

SSL really means TLS
The Kafka connector (and Apache Kafka) actually implements TLS which supersedes the SSL protocol. For historical reasons, Kafka calls this SSL rather than TLS even though the implementation is really TLS. We do the same in order to avoid confusion for experienced Kafka users. That said, for the security-minded audience, it's really TLS!

See the following pages for specific instructions on how to create a connection to Apache Kafka using the various security protocols:

• How to set up SASL Authentication with Kafka
• How to set up Mutual TLS Authentication with Kafka
• How to set up TLS Encryption with Kafka

SASL Properties

When configuring a Kafka connection to use SASL Authentication, the following connection properties are used:

TLS Properties

When configuring a Kafka connection to use TLS or mTLS, the following connection properties are used:

Avro

A popular choice for data serialization on Kafka is Apache Avro . Decodable supports both plain Avro, as well as Avro with a schema registry.

Plain Avro

In order to use plain Avro, the connection property format=avro is required. In this mode, the Avro schema is generated from the connection schema, and is available on the inferred_schema connection property. For example:

decodable connection get 69caa7e1
avro_test_source
 id 69caa7e1
 description A Kafka source connection with plain Avro schema
 connector kafka
 type source
 stream id 865e8555
 schema
 0 field1 INT NOT NULL
 1 field2 DOUBLE NOT NULL
 2 field3 STRING NOT NULL
 properties
 bootstrap.servers kafka-kafka-brokers.kafka.svc.cluster.local:9092
 format avro
 inferred_schema {"type":"record","name":"record","fields":[{"name":"field1","type":"int"},{"name":"field2","type":"double"},{"name":"field3","type":"string"}]}
 topic avro_topic
 target state RUNNING
 actual state RUNNING
 create time 2022-02-04T19:05:19Z
 update time 2022-02-08T17:04:46Z

Avro with Schema Registry

To use Avro with a schema registry, the connection property format=confluent-avro is required. In this mode, the Avro schema is still derived from the connection schema, but it is validated and registered against the configured schema registry.

When using this mode, the following properties are used:

For example, you can create a source Kafka connection using both SASL Authentication, as well as a secured schema registry using:

decodable connection create \
--name avro_registry_source \
--description "A Kafka source connection with SASL auth and a schema registry" \
--type source \
--stream-id 295e2a7f \
--connector kafka \
--prop format="avro-confluent" \
--prop topic="my-topic" \
--prop bootstrap.servers="some.broker.cloud:9092" \
--prop security.protocol=SASL_SSL \
--prop sasl.username="my-api-key" \
--prop sasl.password="my-secret-key" \
--prop sasl.mechanism="PLAIN" \
--prop avro-confluent.schema-registry.url="https://my.schema.registry.cloud" \
--prop avro-confluent.basic-auth.user-info="my-registry-user:my-registry-password" \
--prop avro-confluent.basic-auth.credentials-source="USER_INFO "

Data Type Mapping for Protobuf data

If you are using the Apache Kafka connector to read or write Protobuf data, refer to the following table for how Decodable types map to Protobuf types.