Redpanda is a streaming platform for mission critical workloads. It is Apache® Kafka® compatible and does not require Zookeeper®, JVM, nor any changes to your code. Redpanda is a real-time streaming engine for modern applications, from the enterprise to the solo dev prototyping a React application on her laptop. Redpanda goes beyond the Kafka protocol, into the future of streaming with inline WASM transforms and geo-replicated hierarchical storage. It is a new platform that scales with you from the smallest projects to petabytes of data distributed across the globe.

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kafka connectors can be used in either role, performing as either a source or a sink. The configuration is almost identical, except that source connectors allow for defining a policy for handling parse errors of the incoming data stream.

Configure As A Source

To create and configure a connector for Redpanda, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

1. Select a connector type, either source or sink, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system.

2. Provide a list of one or more Redpanda brokers to use to bootstrap connections. Brokers are specified as hostname/port pair, separated by a colon (e.g., broker-hostname-a:9092). Multiple brokers must be separated by commas with no spaces.

3. Provide a semicolon-separated list of one or more topic names to read data from when the connection type is source, or to write data to when the connection type is sink.

4. Select a broker security protocol, which can be one of the following: PLAINTEXT, TLS, SASL_SSL, or SASL_PLAINTEXT. For more information about Kafka security, see the reference section above.

   • When using TLS, you must also provide:
     • the broker's public certificate, which is used to encrypt traffic from Decodable to external Kafka brokers
     • the client certificate type used to authenticate Kafka clients in Decodable, which can be one of the following: none, SELF_SIGNED, or CSR
     • For mutual TLS authentication, specifying the type is required
   • When using SASL_SSL or SASL_PLAINTEXT, you must also provide:
     • the SASL mechanism, which can be one of the following: PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512
     • the SASL username
     • the SASL password

5. Select a data format used to deserialize and serialize the keys and values, which can be one of the following:

   • JSON, the JSON format allows to read and write JSON data that is based on a JSON schema.
   • Raw, the Raw format allows to read and write raw (byte based) values as a single column.
   • Avro, the Apache Avro format allows to read and write Avro data based on an Avro schema.
   • Debezium (JSON), Debezium provides a unified format schema for changelog and supports serializing messages using JSON.

6. Select a policy for handling record parse errors, which can be either FAIL or IGNORE. When set to FAIL, the connection will change from the Running state to the Stopped state if any validation errors are encountered in the incoming data stream. When set to IGNORE, all validated records will be written, while invalid records will be ignored without stopping the connection or causing any errors/warnings.

For more detailed information about creating and configuring a Redpanda cluster, see the Redpanda Quick Start guide and related documentation.

Configure As A Sink

To create a Kafka connector for use as a sink, all the configuration steps above apply, with the exception of the parse error policy, which is not defined on the outbound end of a data stream.

Reference

Redpanda implements the Kafka API
The Redpanda connector shares all options with the Kafka connector.

Security

Redpanda connections may optionally enable TLS, which encrypts all network traffic between Decodable and the brokers. In addition, the Redpanda connector supports mutual TLS (mTLS) which acts as a bidirectional mode of authentication with the brokers. To configure an mTLS connection, Decodable either provides a self-signed certificate or a certificate signing request (CSR) which you can sign to create a certificate for use with your brokers. See the How To guides for more information.

Properties

The following properties are supported by the Redpanda connector.

SASL Properties

When configuring a Redpanda connection to use SASL Authentication, the following connection properties are used:

TLS Properties

When configuring a Redpanda connection to use TLS or mTLS, the following connection properties are used:

---