Use the Redpanda connector to get data from a Redpanda topic into Decodable. If you are looking for information about how to create a connection to get data into Redpanda from Decodable, see Redpanda in the Connect to a data destination chapter.


Connector name	`redpanda`
Types	`source`, `sink`
Delivery guarantee	at least once or exactly once, based on configuration

Create a connection to Redpanda

Prerequisites

If you want to use TLS or mTLS to secure the connection between Redpanda and Decodable, then you must have a self-signed or third-party signed server certificate and a private key associated with that server certificate.

Because Redpanda is fully compatible with Apache Kafka, the configuration required to ingest data from Redpanda into Decodable is the same as when you ingest data from Kafka. For detailed instructions on how to create a TLS or mTLS secured connection between Redpanda and Decodable, see the following topics.

Steps

Follow these steps to send data from Redpanda into Decodable.

From the Connections page, select the Redpanda connector and complete the following fields.

UI Field	Property Name in the Decodable CLI	Description
Connection Type	N/A	Select Source to use this connector to get data into Decodable.
Bootstrap Servers	`bootstrap.servers`	A comma-separated list of your Kafka brokers. You must enter at least one broker. Enter each broker in the format: :. For example: `broker-a:9092,broker-b:9092` The broker list cannot contain any spaces.
Topic	`topic`	A semicolon-separated list of one or more topic names to read data from.
Broker Security Protocol	`security.protocol`	Specify the security protocol to use when connecting to a broker. Must be one of the following: -PLAINTEXT -TLS: See TLS Properties for information about the additional fields you need to provide when using TLS. -SASL_SSL: See SASL Properties for information about the additional fields you need to provide when using SASL.　 -SASL_PLAINTEXT: See SASL Properties for information about the additional fields you need to provide when using SASL. For more information about the difference between these protocols, see the Security section.
Value Format	`value.format`	The format for data in Redpanda that you want to send to Decodable. Must be one of the following: - JSON - Raw - Avro - Debezium (JSON) If you want to send Change Data Capture (CDC) data through this connector, then you must select Debezium (JSON).
Parse error policy	`parse-error-policy`	Select the behavior that you want when Decodable detects a validation error in the incoming data. When set to `FAIL`, the connection will be stopped. When set to `IGNORE`, the connection continues to run and send records, however, invalid records are ignored.
Consumer Group ID	`properties.group.id`	Optional. Name of the Redpanda consumer group that you want to get data from.
Scan Startup Mode	`scan.startup.mode`	The offset mode that Decodable uses to determine where to start consuming data. Must be one of the following: - group offset: Start from committed offsets in ZK / Kafka brokers of a specified consumer group. You must specify the consumer group in the Consumer Group ID or `properties.group.id` field. - earliest offsets: Start from the earliest offset possible. - latest offset: Start from the latest offset. - timestamp: Start from user-supplied timestamp for each partition. - specific offsets: Start from user-supplied specific offsets for each partition.
Consumer Offset Reset Strategy	`properties.auto.offset.reset`	Select the behavior that you want when there is no initial offset in Redpanda (e.g. a new consumer group) or if the specified offset does not exist (e.g. that data has been deleted).

Select the stream that you’d like to connect to this connector. Then, select Next.
Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Import Schema if you want to paste the schema in the form of an Avro or JSON array.
1. The stream's schema must match the schema of the data that you plan on sending through this connection.
2. For more information about creating a stream or defining the stream schema, see Create a Stream.
Select Next when you are finished providing defining the connection’s schema.
Give the newly created connection a Name and Description and select Save.

Reference

Security

The following table describes the different security protocols that Decodable supports when connecting to Redpanda.

Security protocol	Description
SASL-authenticated	Username and password authentication is used with SASL. Both SSL/TLS and PLAINTEXT encryption is supported, as well as SCRAM and PLAIN authentication mechanisms.
mTLS-authenticated	Two-way SSL authentication is used, so that Decodable and the Redpanda broker(s) authenticate each other using the SSL protocol. Additionally, the connection is encrypted using SSL. To continue an mTLS connection, Decodable provides a self-signed certificate or a certificate signing request (CSR) which you can sign to create a certificate for use with your brokers.
TLS-authenticated	One-way SSL authentication is used. The client (Decodable) holds the server's (Redpanda brokers) public certificate. Data from the Redpanda broker(s) is encrypted using the server's private key and Decodable can decrypt it using the public certificate. Data from Decodable is encrypted using the public certificate and can only be decrypted using the Redpanda broker's private key.
Unauthenticated	No authentication takes place between Decodable and the Redpanda broker(s). The connection is not encrypted.

SASL Properties

When configuring a Redpanda connection to use SASL Authentication, the following connection properties are used.

UI Field	Property Name in the Decodable CLI	Description
`sasl.mechanism`	`sasl.mechanism`	Specify the SASL mechanism as configured by the Redpanda broker. Support values are ["PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512"].
`sasl.username`	`sasl.username`	The provided username or API key for authentication.
`sasl.password`	`sasl.password`	The ID of the secret resource containing the password. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run `decodable secret list` to view available secrets or `decodable secret --help` for help with creating a new secret. Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Manage Roles, Groups, and Permissions for more information.

TLS Properties

When configuring a Redpanda connection to use TLS or mTLS, the following connection properties are used.

UI Field	Property Name in the Decodable CLI	Description
Broker Certificate	`tls.broker.certificate`	Broker's public certificate used to encrypt traffic from Decodable to external Redpanda brokers. The property is only used when security.protocol=TLS.
TLS Client Certificate	`tls.client.certificate`	Specify the client certificate used by Decodable to connect to external Redpanda brokers with mTLS.
TLS Client Key	`tls.client.key`	The ID of the secret resource containing the client key. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run `decodable secret list` to view available secrets or `decodable secret --help` for help with creating a new secret. Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Manage Roles, Groups, and Permissions for more information.
N/A	`tls.broker.signed_client_certificate`	Specify the client certificate signed by the broker. This property is required when `tls.client.certificate.type=CSR`.

Apache Kafka, Kafka®, Apache® and associated open source project names are either registered trademarks or trademarks of The Apache Software Foundation.