How to: Set up a connection to Apache Kafka using Upstash

This guide goes through the required steps to connect Decodable to an Upstash Kafka cluster.

Upstash provides Kafka as a serverless service, enabling users to use Kafka without the need for paying and managing their own servers and cluster.

Prerequisites

Before you send data from an Upstash Kafka cluster into Decodable, you must have an Upstash Kafka cluster created. See Getting Started in the Upstash documentation for instructions on how to create an Upstash Kafka cluster.

Steps

Do the following steps to send data from an Upstash Kafka cluster to Decodable.

  1. From the Connections page, select Apache Kafka and complete the following required fields.

    Field What to enter

    Connection Type

    Source

    Bootstrap Servers

    A comma-separated list of host and port pairs representing the addresses of the Kafka brokers to connect to. For example: tops-stingray-7863-us1-kafka.upstash.io:9092

    Topic

    The name of the topic that you want to get data from. You can find your topic names in the Upstash console.

    Broker Security Protocol

    SASL_SSL

    SASL Username

    The username that you use to login to your Upstash Kafka cluster.

    SASL Password

    The secret containing the password that you use to login to your Upstash Kafka cluster.

    SASL Mechanism

    SCRAM-SHA-256

    Value Format

    JSON

    The other fields are optional. See Apache Kafka for information about the optional fields you can configure. Select Next when you are done.

  2. Select the stream that you’d like to connect to this connector. This will be the stream that receives events from Upstash. Then, select Next.

  3. If you selected a new stream, you must define the schema of the stream. Select New Schema to manually enter the fields and field types present or Import Schema if you want to paste the schema in the form of a JSON array.

    1. The stream’s schema must match the schema of the data that you plan on sending through this connection.

  4. Select Next when you are finished providing defining the connection’s schema.

  5. Give the newly created connection a Name and Description and select Save.

You can now ready to start sending data from an Upstash Kafka cluster into a Decodable stream!