Redpanda sink connector

Use the Redpanda connector to get data from a Redpanda topic into Decodable. If you are looking for information about how to create a connection to get data into Redpanda from Decodable, see Redpanda source connector.

Features

Delivery guarantee

At least once or exactly once, based on configuration

Prerequisites

If you want to use TLS or mTLS to secure the connection between Decodable and Redpanda, then you must have a self-signed or third-party signed server certificate and a private key associated with that server certificate.

Because Redpanda is fully compatible with Apache Kafka, the configuration required to process data from Redpanda into Decodable is the same as when you process data from Kafka. For detailed instructions on how to create a TLS or mTLS secured connection between Redpanda and Decodable, see the following topics.

Steps

If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is redpanda.
  1. From the Connections page, select the Redpanda connector and complete the following fields.

    UI Field

    Property Name

    Description

    Connection Type

    N/A

    Select Sink to use this connector to get data into Decodable.

    Bootstrap Servers

    bootstrap.servers

    A comma-separated list of your Kafka brokers. You must enter at least one broker. Enter each broker in the format: :. For example: broker-a:9092,broker-b:9092

    The broker list can’t contain any spaces.

    Topic

    topic

    A semicolon-separated list of one or more topic names to read data from.

    Broker Security Protocol

    security.protocol

    Specify the security protocol to use when connecting to a broker. Must be one of the following:

    • PLAINTEXT

    • TLS: See TLS properties for information about the additional fields you need to provide when using TLS.

    • SASL_SSL: See SASL properties for information about the additional fields you need to provide when using SASL.

    • SASL_PLAINTEXT: See SASL properties for information about the additional fields you need to provide when using SASL.

    For more information about the difference between these protocols, see the Security section.

    Value Format

    value.format

    The format for data in Redpanda that you want to send to Decodable. Must be one of the following:

    • JSON

    • Raw

    • Avro

    • Debezium (JSON): If you want to send Change Data Capture (CDC) data through this connector, then you must select Debezium (JSON).

    Key Format

    key.format

    The format used to deserialize and serialize the key part of Redpanda messages. Only applicable when value.format=Debezium(JSON).

  2. Select the stream that you’d like to connect to this connector. Then, select Next.

  3. Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Import Schema if you want to paste the schema in the form of an Avro or JSON array.

    1. The stream’s schema must match the schema of the data that you plan on sending through this connection.

    2. For more information about creating a stream or defining the stream schema, see Create and manage Streams.

  4. Select Next when you are finished providing defining the connection’s schema.

  5. Give the newly created connection a Name and Description and select Save.

Security

Redpanda connections can optionally enable TLS, which encrypts all network traffic between Decodable and the brokers. In addition, the Redpanda connector supports mutual TLS (mTLS) which acts as a bidirectional mode of authentication with the brokers. To configure an mTLS connection, Decodable either provides a self-signed certificate or a Certificate Signing Request (CSR) which you can sign to create a certificate for use with your brokers. See the How To guides for more information.

SASL properties

When configuring a Redpanda connection to use How to set up SASL Authentication with Kafka, the following connection properties are used.

UI Field Property Name Description

SASL Mechanism

sasl.mechanism

Specify the SASL mechanism as configured by the Redpanda broker. Supported values are:

  • PLAIN

  • SCRAM-SHA-256

  • SCRAM-SHA-512

SASL Username

sasl.username

The provided username or API key for authentication.

SASL Password

sasl.password

The ID of the secret resource containing the password. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Roles, groups, and permissions for more information."

TLS properties

When configuring a Redpanda connection to use mTLS or TLS, the following connection properties are used.

UI Field Property Name Description

Broker Certificate

tls.broker.certificate

The broker’s public certificate used to encrypt traffic between external Redpanda brokers and Decodable. This property is only used when security.protocol=TLS.

TLS Client Certificate

tls.client.certificate

Specify the client certificate used by Decodable to connect to external Redpanda brokers with mTLS.

TLS Client Key

tls.client.key

The ID of the secret resource containing the client key. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Roles, groups, and permissions for more information."

N/A

tls.broker.signed_client_certificate

Specify the client certificate signed by the broker. This property is required when tls.client.certificate.type=CSR.

Connector starting state and offsets

A new sink connection will start reading from the Latest point in the source Decodable stream. This means that only data that’s written to the stream when the connection has started will be sent to the external system. You can override this when you start the connection to Earliest if you want to send all the existing data on the source stream to the target system, along with all new data that arrives on the stream.

When you restart a sink connection it will continue to read data from the point it most recently stored in the checkpoint before the connection stopped. You can also opt to discard the connection’s state and restart it afresh from Earliest or Latest as described above.

Learn more about starting state here.