Redpanda
Redpanda is a streaming platform for mission critical workloads. It is Apache® Kafka® compatible and does not require Zookeeper®, JVM, nor any changes to your code. Redpanda is a real-time streaming engine for modern applications, from the enterprise to the solo dev prototyping a React application on her laptop. Redpanda goes beyond the Kafka protocol, into the future of streaming with inline WASM transforms and geo-replicated hierarchical storage. It is a new platform that scales with you from the smallest projects to petabytes of data distributed across the globe.
Getting Started
Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kafka connectors can be used in either role, performing as either a source
or a sink
. The configuration is almost identical, except that source connectors allow for defining a policy for handling parse errors of the incoming data stream.
Configure As A Source
To create and configure a connector for Redpanda, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection
, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.
-
Select a connector type, either
source
orsink
, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system. -
Provide a list of one or more Redpanda brokers to use to bootstrap connections. Brokers are specified as hostname/port pair, separated by a colon (e.g., broker-hostname-a:9092). Multiple brokers must be separated by commas with no spaces.
-
Provide a semicolon-separated list of one or more topic names to read data from when the connection type is
source
, or to write data to when the connection type issink
. -
Select a broker security protocol, which can be one of the following:
PLAINTEXT
,TLS
,SASL_SSL
, orSASL_PLAINTEXT
. For more information about Kafka security, see the reference section above.- When using TLS, you must also provide:
- the broker's public certificate, which is used to encrypt traffic from Decodable to external Kafka brokers
- the client certificate type used to authenticate Kafka clients in Decodable, which can be one of the following:
none
,SELF_SIGNED
, orCSR
- For mutual TLS authentication, specifying the type is required
- When using SASL_SSL or SASL_PLAINTEXT, you must also provide:
- the SASL mechanism, which can be one of the following:
PLAIN
,SCRAM-SHA-256
, orSCRAM-SHA-512
- the SASL username
- the SASL password
- the SASL mechanism, which can be one of the following:
- When using TLS, you must also provide:
-
Select a data format used to deserialize and serialize the keys and values, which can be one of the following:
JSON
, the JSON format allows to read and write JSON data that is based on a JSON schema.Raw
, the Raw format allows to read and write raw (byte based) values as a single column.Avro
, the Apache Avro format allows to read and write Avro data based on an Avro schema.Debezium (JSON)
, Debezium provides a unified format schema for changelog and supports serializing messages using JSON.
-
Select a policy for handling record parse errors, which can be either
FAIL
orIGNORE
. When set to FAIL, the connection will change from theRunning
state to theStopped
state if any validation errors are encountered in the incoming data stream. When set to IGNORE, all validated records will be written, while invalid records will be ignored without stopping the connection or causing any errors/warnings.
For more detailed information about creating and configuring a Redpanda cluster, see the Redpanda Quick Start guide and related documentation.
Configure As A Sink
To create a Kafka connector for use as a sink
, all the configuration steps above apply, with the exception of the parse error policy, which is not defined on the outbound end of a data stream.
Reference
Connector name | redpanda |
Types | source , sink |
Delivery guarantee | at least once or exactly once, based on configuration |
Redpanda implements the Kafka API
The Redpanda connector shares all options with the Kafka connector.
Security
Redpanda connections may optionally enable TLS, which encrypts all network traffic between Decodable and the brokers. In addition, the Redpanda connector supports mutual TLS (mTLS) which acts as a bidirectional mode of authentication with the brokers. To configure an mTLS connection, Decodable either provides a self-signed certificate or a certificate signing request (CSR) which you can sign to create a certificate for use with your brokers. See the How To guides for more information.
Properties
The following properties are supported by the Redpanda connector.
Property | Disposition | Description |
---|---|---|
bootstrap.servers | required | A comma-separated list of Kafka brokers to use to bootstrap connections. Each broker must be a colon-separated (: ) hostname/port pair. The list must not include any spaces. Example: broker-a:9092,broker-b:9092 . Kafka brokers must be able to receive connections from Decodable's network blocks. |
topic | required | A semicolon separated list of topic names to read data from when connection type is source or write data to when connection type is sink. |
format | optional; if not specified, value.format must be used | Specifies a data format used to deserialize and serialize the value. Supported formats are ["raw", "json", "avro", "confluent-avro"]. |
value.format | optional; if not specified, format must be used | An alias for format . |
key.format | optional | Specifies a data format used to deserialize and serialize the key. Supported formats are ["raw", "json", "avro", "confluent-avro"]. |
security.protocol | optional | Specifies the protocol used to communicate with brokers. Valid values are: TLS, SASL_PLAINTEXT, SASL_SSL. If not specified, data will transfer in plain text. |
SASL Properties
When configuring a Redpanda connection to use SASL Authentication, the following connection properties are used:
Property | Disposition | Description |
---|---|---|
sasl.mechanism | required | Specify the SASL mechanism as configured by the Redpanda broker. Support values are ["PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512"]. |
sasl.username | required | The provided username or API key for authentication. |
sasl.password | required | The provided password or API secret key for authentication. |
TLS Properties
When configuring a Redpanda connection to use TLS or mTLS, the following connection properties are used:
Property | Disposition | Description |
---|---|---|
tls.broker.certificate** | required | Broker's public certificate used to encrypt traffic from Decodable to external Redpanda brokers. The property is only used when security.protocol=TLS. |
tls.client.certificate.type | required | Specifies the client certificate type used to authenticate Redpanda clients in Decodable. This property is required when configuring mTLS. Valid values are ["CSR", "Self-Signed"]. The property is only used when security.protocol=TLS. |
tls.broker.signed_client_certificate | required | Specifies the client certificate signed by the broker. This property is required when tls.client.certificate.type=CSR . |
Updated 9 months ago