AWS Kinesis

Use the AWS Kinesis Connector to get data from Amazon Kinesis into Decodable. If you are looking for information about how to send data from Decodable into Amazon Kinesis, see AWS Kinesis in the Connect to a data destination chapter.

Overview

Connector name

kinesis

Type

source, sink

Delivery guarantee

at least once

Prerequisites

Before you can get data from Amazon Kinesis, the following requirements must be met:

  • You must have an Identity and Access Management (IAM) user with the necessary permissions for reading from the Kinesis stream.

  • You must also have the access key ID and secret access key for that IAM user.

AWS permissions for getting data from Amazon Kinesis Streams

AWS IAM uses an ExternalId to grant third-parties like Decodable access to your AWS resources. In order for Decodable to connect to your Kinesis stream, you will need:

  • An IAM Role with two policies:

    • An AssumeRole Trust Policy allowing access from Decodable’s AWS account. Make sure that the ExternalId matches your unique Decodable account name and that the Principal is 671293015970. This is Decodable’s AWS Account ID. The following snippet is an example of a trust policy. You can replace <My_Decodable_Account> with the name of your unique Decodable account.

      • Make sure you are supplying your unique Decodable account name to the Trust Policy. This is different from the account’s display name. Your unique Decodable account name will be all lowercase and contain no spaces.

        {
        "Version": "2012-10-17",
        "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::671293015970:root"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringEquals": {
              "sts:ExternalId": "<My_Decodable_Account>"
            }
          }
        }
        ]
        }
    • A Permissions Policy with the needed permissions on your stream. The following snippet is an example of a permissions policy that allows full access to your Kinesis stream. You can also restrict the permissions for specific Kinesis streams by specifying a Kinesis ARN instead of the wildcard *. You can also list out specific action permissions. For more information, see Controlling Access to Amazon Kinesis Data Streams Resources Using IAM.

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "kinesis:*",
            "Resource": "*"
          }
        ]
      }
  • The ARN of this Role. Decodable will ask you for the ARN of your IAM Role as part of connection creation.

Our servers will assume that role by using an ExternalId value matching your unique Decodable account name. We’ll use that to talk to your bucket.

The values of your ExternalId, Role ARN, or bucket names are not treated as a secret by Decodable or AWS. Anyone with permissions to view these values will be able to see them.

Steps

Follow these steps to get data from Kinesis into Decodable. These steps assume that you are using the Decodable web interface. However, if you want to use the Decodable CLI to create the connection, you can refer to the Property Name column for information about what the underlying property names are.

  1. From the Connections page, select AWS Kinesis and complete the following fields.

    UI Field Property Name in the Decodable CLI Description

    Connection Type

    N/A

    Select Source to use this connector to get data into Decodable.

    Stream Name

    stream

    The name of the stream that you want to receive data from.

    AWS Region

    aws.region

    The region that the stream is located in.

    IAM Role ARN

    aws.credentials.role.arn

    The AWS ARN of the IAM Role that you configured as part of the prerequisites.

    Value Format

    format

    The format of the data in the stream. Must be json.

  2. Select the stream that you’d like to connect to this connector. Then, select Next.

  3. Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Import Schema if you want to paste the schema in Avro or JSON format.

    1. The stream’s schema must match the schema of the data that you plan on sending through this connection.

    2. For more information about creating a stream or defining the stream schema, see Create and Manage Streams.

  4. Select Next when you are finished providing defining the connection’s schema.

  5. Give the newly created connection a Name and Description and select Save.

Reference

Extract message metadata

When consuming records from Amazon Kinesis into Decodable, you can access specific metadata fields associated with each Kinesis record. A metadata field is identified by a string-based key and an associated data type. For example, the Amazon Kinesis connector has a metadata field with the key timestamp and data type TIMESTAMP_LTZ(3). See Available metadata for a list of metadata fields that you have access to.

A metadata field has its type formed as:

{datatype} METADATA [FROM '{key}']

Thus, a metadata field is indicated by the METADATA keyword, with an optional FROM to provide the key. If the key is not provided explicitly with FROM, it defaults to the name of the field.

For example:

TIMESTAMP_LTZ(3) METADATA FROM 'timestamp'

Available metadata

When you connect to Amazon Kinesis as a data source, you have access to the following metadata fields.

Key Data Type Description

timestamp

TIMESTAMP_LTZ(3) NOT NULL

The approximate time when the record was inserted into the stream.

shard-id

VARCHAR(128) NOT NULL

The unique identifier of the shard within the stream from which the record was read.

sequence-number

VARCHAR(128) NOT NULL

The unique identifier of the record within its shard.