About Streams

A stream is a sequence of data records that are stored in a distributed manner and processed in real-time by Decodable. Streams can be read or written to, and they always have a schema that specifies its fields. Once a stream is defined, it can be used as an input or output by any number of pipelines or connections. If you are familiar with Apache Kafka or Amazon Kinesis, a stream in Decodable is conceptually similar to a "stream" in Kinesis or a "topic" in Apache Kafka.

Stream types

The following table describes the different types of streams that Decodable supports.

Stream type Description

Stream type	Description
Append	A type of data stream in which new records are continuously added to the end of the stream. In other words, an append stream is a stream that’s only ever appended to and each individual record in an append stream is an independent event. Partitions When data is written to an append stream, it’s split into multiple partitions based on the partition key specified by the user. Decodable ensures that records with the same partition key are stored in the same partition and processed by the same task. To specify a partition key, identify one or more existing physical fields in the schema. If you don’t specify any fields to use as the partition key, then batches of records are distributed evenly over all stream partitions. To create a stream with the `APPEND` type, define a schema without specifying a primary key. For more information about how to create a stream, see Create and manage Streams.
Change	A type of data stream that captures changes to a database in real-time. Records in a change stream represent a change made to a database, such as updates, inserts, and deletes. A record in a change stream is called a Change Data Capture (CDC) record. To create a stream with the `CHANGE` type, you must specify one or more fields in the stream’s schema to use as the primary key. A primary key is one or more fields that contain a value that can be used to uniquely identify each row in a table. Decodable also supports a composite primary key, which is when you specify more than one field to use as a primary key that, when combined, can be used to uniquely identify each row in the table. For more information about how to create a stream, see Create and manage Streams.

Append

A type of data stream in which new records are continuously added to the end of the stream. In other words, an append stream is a stream that’s only ever appended to and each individual record in an append stream is an independent event.

Partitions

When data is written to an append stream, it’s split into multiple partitions based on the partition key specified by the user. Decodable ensures that records with the same partition key are stored in the same partition and processed by the same task. To specify a partition key, identify one or more existing physical fields in the schema. If you don’t specify any fields to use as the partition key, then batches of records are distributed evenly over all stream partitions.

To create a stream with the APPEND type, define a schema without specifying a primary key. For more information about how to create a stream, see Create and manage Streams.

Change

A type of data stream that captures changes to a database in real-time. Records in a change stream represent a change made to a database, such as updates, inserts, and deletes. A record in a change stream is called a Change Data Capture (CDC) record.

To create a stream with the CHANGE type, you must specify one or more fields in the stream’s schema to use as the primary key. A primary key is one or more fields that contain a value that can be used to uniquely identify each row in a table. Decodable also supports a composite primary key, which is when you specify more than one field to use as a primary key that, when combined, can be used to uniquely identify each row in the table.

For more information about how to create a stream, see Create and manage Streams.

You can also convert an append stream to a change stream and vice versa with the to_change() and to_append() functions. See Process a change stream as an append stream and Process an append stream as a change stream for an example.