How To: Set up a connection to Confluent Cloud

Decodable supports reading and writing data into Confluent Cloud Kafka topics. This is enabled in the web app using the "Confluent Cloud" connector type.

Set up the Decodable connection

Connection type

  • Click "Connections" in the nav bar

  • Click the "New Connection" button on the top right

  • Choose source or sink then select the "Confluent Cloud" connector type and click "Next"

Connect to Confluent Cloud cluster

  • The cluster id and API endpoint can be found on the Cluster overview -> Cluster settings page, in the Identification box.

This image shows the Cluster settings page under the Cluster overview tab. On the Cluster settings page

  • API keys and secrets can be created in Confluent Cloud by clicking Data Integration -> API keys. Here you can see any API keys you’ve already created or create a new one with the "Add key" button. Make sure you save the API key and secret pair somewhere you can find it since you won’t be able to get the secret again through the UI

This image shows the API keys page under the Data integration tab. On the API keys page

  • Click "Next"

Choose a topic

  • Now select the topic you want to connect using the auto-complete text field

  • If you’re using a newly created API key, you may not be seeing any topics yet because it takes a minute or two for Confluent to propagate the new key through the system. Try waiting a bit and clicking "Refresh Topics"

Select a stream to connect to

  • Either select the existing Decodable stream this connection will be connected to or click "New Stream" to create a new stream for the connection

  • If you’re connecting to an existing stream, you can verify the schema and rename fields, but not add or remove any fields

  • If you’re creating a new stream you have three options for defining your schema

    1. Manually define a new schema by adding fields and giving them types

    2. Select "Structured Schema Definition" from the drop down to upload a schema file that you already have. Select the appropriate type of schema from the "Schema Type" dropdown, then type or paste the schema definition into the "Schema" text box.

    3. Select "Schema Registry" to connect to your Confluent Cloud schema registry and get the schema from there

  • You can get the required information by clicking "Schema Registry" in the Confluent Cloud sidebar.

This image highlights the Schema Registry option in the sidebar. height=200

  • Copy the API endpoint from the API endpoint box

  • API key/secret pairs can be created in the API credentials box. Just like for the cluster, make sure to save these because you won’t be able to access the secrets from the UI again.

This image shows that an API key and secret was generated. The values of the key and secret are shown once after creation.

Finish

  • Give your connection a name and (optionally) a description, then click "Create Connection"

  • Be sure to activate your connection to get the data flowing!