About connections

One of the first steps in incorporating Decodable to your existing data ecosystem is by creating connections in Decodable to get data in from a data source or send data out to a data destination. Decodable includes connectors that provide read and write support for a variety of data sources and destinations including Amazon Kinesis source connector, Amazon S3 source connector, Apache Kafka source connector, Apache Pulsar sink connector, Snowflake sink connector, and more. To allow Decodable to access your data, you must configure a connection that contains your credentials for the data source and destination.

Previously, Decodable operated under a model where each connection was tied to a single stream. This approach resulted in a strict 1:1 relationship between connections and streams. Consequently, managing data from multiple tables within a database posed challenges, leading to the need to create a separate connection for each table.

To address these challenges and provide greater flexibility, Decodable is moving away from the traditional 1:1 paradigm. This evolution enables both the ingestion of data from multiple tables and egress of data into multiple tables using a single connection. While the implementation of this feature is gradually expanding across our connector library, connectors like the MySQL CDC source and the Snowflake sink are among the early adopters, supporting this new behavior. As an example, with this updated functionality, you can now ingest data from multiple MySQL tables and databases on the same host using a single MySQL connection, which can route data to multiple streams within Decodable.

After creating a connection to your data source or destination, activate it to start streaming data to or from the Decodable platform. A connection to a data source is also known as a source connection and a connection to a data destination is also known as a sink connection.

Decodable creates network connections to the resources you specify in connections. As a result, things like host names must be resolvable, and IP addresses must be routable. See Available regions and IP addresses if you want to connect to a third-party service that requires explicit whitelisting to enable external access. Let us know if you need help managing connectivity to your data infrastructure.

For instructions on how to create a connection to a specific data source or destination, refer to the topic in the Connect to a data source or Connect to a data destination chapters that correspond to the data location that you are interested in.