Snowflake

Snowflake is a fully managed software-as-a-service (SaaS) that provides a single platform for data warehousing, data lakes, data engineering, data science, data application development, and secure sharing and consumption of real-time / shared data. Snowflake features out-of-the-box features like separation of storage and compute, on-the-fly scalable compute, data sharing, data cloning, and third-party tools support in order to handle the demanding needs of growing enterprises.

The Snowflake data platform is not built on any existing database technology or “big data” software platforms such as Hadoop. Instead, Snowflake combines a completely new SQL query engine with an innovative architecture natively designed for the cloud. To the user, Snowflake provides all the functionality of an enterprise analytic database, along with many additional special features and unique capabilities. Snowflake runs completely on cloud infrastructure. All components of Snowflake’s service (other than optional command line clients, drivers, and connectors), run in public cloud infrastructures. Snowflake uses virtual compute instances for its compute needs and a storage service for persistent storage of data. Snowflake cannot be run on private cloud infrastructures (on-premises or hosted) as it is not a packaged software offering that can be installed by a user. Snowflake manages all aspects of software installation and updates.

Getting Started

Sending a Decodable data stream to Snowflake is accomplished in two stages, first by creating an AWS S3 sink connector, and then by using that as a data source in Snowflake.

Configure As A Sink

This example demonstrates using Amazon S3 as the sink from Decodable and the source for Snowflake. Sign in to the Decodable Web Console and follow the configuration steps provided for the Amazon S3 Connector to create a sink connector. For examples of using the command line tools or scripting, see the How To guides.

Create S3 Data Source

To ingest a data stream into Snowflake from S3, Amazon SQS (Simple Queue Service) notifications are used to trigger Snowpipe data loads.

  1. Create an S3 sink in Decodable
  2. Configure Snowpipe following the steps outlined in the Decodable example
  3. Create an Append Stream in Snowflake
  4. Create a Task to Merge changes into the Mirror Table
  5. View Tasks

For more detailed information, please refer to the Snowflake example in the Decodable GitHub repository and the Automating Snowpipe for Amazon S3 documentation.

Reference

Connector namesnowflake
Typesink

Additional Snowflake Support
If you are interested in a direct Decodable Connector for Snowflake, please contact [email protected] or join our Slack community and let us know!