How To: Set up a connection to Confluent Cloud
Decodable supports reading and writing data into Confluend Cloud Kafka topics. This is enabled in the web app using the "Confluent Cloud" connector type
Set up the Decodable Connection
- Click "Connections" in the nav bar
- Click the "New Connection" button on the top right
- Choose source or sink then select the "Confluent Cloud" connector type and click "Next"
Connect to Confluent Cloud Cluster
- The cluster id and API endpoint can be found on the Cluster overview -> Cluster settings page, in the Identification box
- API keys and secrets can be created in Confluent Cloud by clicking Data Integration -> API keys. Here you can see any API keys you've already created or create a new one with the "Add key" button. Make sure you save the API key and secret pair somewhere you can find it since you won't be able to get the secret again through the UI
- Click "Next"
Choose a topic
- Now select the topic you want to connect using the auto-complete text field
- If you're using a newly created API key, you may not be seeing any topics yet because it takes a minute or two for Confluent to propagate the new key through the system. Try waiting a bit and clicking "Refresh Topics"
Select a stream to connect to
- Either select the existing Decodable stream this connection will be connected to or click "New Stream" to create a new stream for the connection
- If you're connecting to an existing stream, you can verify the schema and rename fields, but not add or remove any fields
- If you're creating a new stream you have three options for defining your schema
- Manually define a new schema by adding fields and giving them types
- Select "Structured Schema Definition" from the drop down to upload a schema file that you already have. Select the appropriate type of schema from the "Schema Type" dropdown, then type or paste the schema definition into the "Schema" text box.
- Select "Schema Registry" to connect to your Confluent Cloud schema registry and get the schema from there
- You can get the required information by clicking "Schema Registry" in the Confluent Cloud sidebar
- Copy the API endpoint from the API endpoint box
- API key/secret pairs can be created in the API credentials box. Just like for the cluster, be sure to save these because you won't be able to access the secrets from the UI again
- Give your connection a name and (optionally) a description, then click "Create Connection"
- Be sure to activate your connection to get the data flowing!
Updated about 1 year ago