Connecting Decodable to sources and destinations

One of the first steps in incorporating Decodable to your existing data ecosystem is to create connections in Decodable to get data in from a data source or send data out to a data destination. Decodable includes connectors that provide read and write support for a variety of data sources and destinations including Amazon Kinesis, Amazon S3, Apache Kafka, Apache Pulsar, Snowflake, and more.

After creating a connection to your data source or destination, activate it to start streaming data to or from the Decodable platform. A connection to a data source is also known as a source connection and a connection to a data destination is also known as a sink connection.

Streams per Connector

Decodable connectors support one of the following two models:

  • One connector instance can read/write from one stream.

    For example, one Kafka source connector can read from one Kafka topic and write to one Decodable stream.

or

  • One connector instance can read/write from multiple streams.

    For example:

    • One MySQL source connector can read from multiple tables in a database, writing to multiple Decodable streams.

    • One Snowflake sink connector can read from multiple Decodable streams and write to multiple tables on a Snowflake database

If a connector supports multiple streams it will be indicated under Features on the connector’s reference page.

Network connectivity

Decodable creates network connections to the resources you specify in connections. As a result, things like host names must be resolvable, and IP addresses must be routable. See Available regions and IP addresses if you want to connect to a third-party service that requires explicit whitelisting to enable external access.

Decodable supports AWS PrivateLink for connectivity to private services, including Amazon RDS and Managed Service for Kafka (MSK). AWS PrivateLink creates a connection from Decodable to private services in AWS without exposing them on the public Internet.

Contact Decodable support for help managing connectivity to your data infrastructure, or to configure PrivateLink for your connections.

For instructions on how to create a connection to a specific data source or destination, refer to the topic in the Connect to a data source or Connect to a data destination chapters that correspond to the data location that you are interested in.

Processing guarantees for Connectors

While Decodable’s stream processing engine always processes data exactly once, some connectors don’t support this level of guarantee or make it configurable.

The processing guarantee varies depending on the connector. Connectors always specify whether they provide at least once or exactly once processing. For the exact processing guarantee, refer to the topic corresponding to the connector that you are interested in.