Create and manage streams

A stream transports records between Decodable components. Once a stream is defined, it can be used as an input or output by any number of pipelines or connections. There are two types of streams: append streams and change streams. See About Streams for more information.

Create a stream using Decodable Web

There are a few different ways to create a stream. Because the schema of a connection must match the schema of the stream that it’s connected to, it’s easier to prevent schema mismatches if you create and configure a stream as part of connection creation. However, you can also create and configure a stream from the Streams page. If you want to create a stream that contains records from pipeline(s), also known as output streams, then you can create it as part of pipeline creation.

It ultimately does not matter which method you use to create a stream, but depending on the use case you are trying to do, it can be easier to start from one place versus another.

Creation method Do this when…​

As part of connection creation

You are setting up a new end-to-end workflow and creating a stream as part of that workflow.

As part of pipeline creation

You want Decodable to create an output stream for you based on the pipeline’s SQL query. This creation method is only valid for output streams, or streams that are receiving data from pipelines.

From the Streams page

You want to connect the stream to multiple connections, or you want to update a stream that is in-use without having to stop data ingestion.

After you create a connection

When you create a connection, you are prompted to select the stream that you’d like the connection to send data to. Perform the following steps to create a new stream.

  1. From the Connections page, select a connector that represents the data source or destination that you want to get data from or send data to. After filling in the required fields for the connector, select Next.

  2. Select New Stream. Give your stream a name and a description. Select Next.

    1. Best practices are to include a description that describes the data that this stream contains.

  3. Specify the schema of the data that you plan to send to this stream. Select Save to continue.

    1. The stream’s schema must match the schema of the connection that it is attached to. All records in a stream must have the same schema.

    2. See Manage Schemas for detailed information on the various options available to you when defining a schema.

  4. Give your connection a Name and Description and select Save.

  5. (Optional) Preview your stream to make sure that data is flowing from your connection to your newly created stream.

    1. Start your connection to get data flowing into the stream by selecting Start. Wait until the status of the connection displays Running.

    2. Select Outbound to stream - Stream Name in the lineage bar to navigate to the detailed view of the stream.

    3. Select Run Preview. When you start a preview session, Decodable starts polling for any data sent to the stream. The preview session continues until you stop it, 2 minutes have passed, or 30 records are sampled, whichever comes first.

      1. Note: Preview shows null for computed fields. However, the correct values will be provided correctly to any pipelines or connections attached to the stream.

      2. If you do not see any records, make sure that the upstream data source that you are connected to is generating data.

You can now use your connection to begin sending data to Decodable, and your stream is listed in the Streams page.

From the streams page

You can also create and configure a stream from the Streams page. Do the following steps to create and define a stream.

  1. From the Streams page, select New Stream.

  2. Select the type of stream that you want to create.

    1. If you want to send individual records, select Append Stream.

      1. For append streams, you can optionally add a partition key by specifying one or more existing physical fields in the schema. If you do not specify any fields to use as the partition key, then records are distributed randomly over all stream partitions.

    2. If you want to send data representing changes (updates, inserts, and deletes) to a database, select Change Stream.

      1. For change streams, you must specify a field to use as a primary key.

      2. For more information about change streams, see About change data capture and change streams.

    If you change your mind about what type of data you want this stream to contain, you can change the stream type with the to_change() and to_append() functions. See Pipelines examples for the most popular data transformations for an example.

  3. Fill out the rest of the page. See Manage Schemas for detailed information on the various options available to you when defining a stream’s schema.

Create a stream using the Decodable CLI

The following example walks you through the steps required to create a stream using the Decodable CLI. You must have the Decodable CLI installed.

Example: Create and update an append Stream

The following example demonstrates how to create an append stream.

Create an append stream

Run the following command to create an append stream called my_stream with one field raw_data with type string.

decodable stream create +
  --name my_stream +
  --description "raw data from prod Kafka" +
  --field raw_data=STRING
# Created stream my_stream (2fc5dc5a)

View the append stream

The following example demonstrates how to view the stream that you just created.

decodable stream get 2fc5dc5a
# my_stream
#   id                       2fc5dc5a
#   description              raw data from prod Kafka
#   schema
#     0  raw_data              STRING
#     1  new_field             STRING
#   create time              2021-11-03T20:28:48Z
#   update time              2021-11-03T21:39:34Z

Updating an append stream

The following example demonstrates updating the append stream to add another field. NOTE: When you are updating or removing fields, you must supply the entire new schema.

decodable stream update 2fc5dc5a \
  --field raw_data=STRING  \
  --field new_field=STRING
# Updated stream "2fc5dc5a"

Example: Creating a change stream

The following example demonstrates how to create a change stream.

decodable stream create +
  --name my_change_stream +
  --description "transactions from MYSQL database" +
  --field "customerID=int not null" --primary-key "customerID" +
  --field "productID=string not null" --primary-key "productID" +
  --field "transaction=int"

# Created stream my_stream (2cd1d83a)

Manage streams

You can use the Streams page to manage your existing streams. Select a stream to open the Overview tab for that stream. From the Overview tab, you can do things like:

  • View the connections or pipelines attached to the stream.

  • Use the lineage bar to trace the flow of data by navigating to the connected pipeline or connection. The lineage bar is the UI element that displays the Inbound from and Outbound to information.

  • Preview what the incoming data looks like.

  • View the schema of a stream by switching to the Schema tab.

You can also Edit, Delete, Clone, or Clear the stream from the More Options (…​) dropdown.

  • Edit: Edit the name and description of a stream.

  • Delete: Delete a stream.

  • Clone: Clone a stream. This clones the stream but does not clone the pipelines, connections, or data associated with the stream.

  • Clear: Clears all existing records from a stream.

For other common stream management tasks, see the following pages: