Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kinesis connectors can be used in either role, performing as either a source or a sink.

Configure As A Source

To create and configure a connector for Kinesis, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

1. Select a connector type, either source or sink, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system.

2. Provide the name of your Kinesis stream.

3. Provide the AWS region of your Kinesis stream.

4. Specify the AWS ARN of the IAM role. For example, arn:aws:iam::111222333444:role/decodable-kinesis-access.

5. The value format will default to JSON, since that is the only option for Kinesis connectors.

For more detailed information about Kinesis, see the Kinesis Getting Started
guide and related documentation.

Note: The AWS Kinesis connector might require task size L tasks when consuming from streams with many shards (e.g. streams that have been created with high data volumes). See Manage task count and sizes.

Configure As A Sink

To create a Kinesis connector for use as a sink, all the configuration steps above apply.

Reference

The Kinesis connector streams data into or out of a Kinesis stream in your AWS account. To use it, configure an AWS IAM Role as described below, with specific permissions to access the stream.

Properties

If you are using the Decodable CLI to create or edit a connection to a Kinesis stream, then use the following table as a reference for what properties are required and supported.

IAM Role, Permissions, and Security

To be secure, you, AWS, and Decodable work together to ensure only Kinesis connections in your Decodable Account can access your Kinesis streams.

How?

AWS IAM provides a special mechanism — called ExternalId — that you and Decodable will use as described here, which ensures access from Decodable to your stream happens only for your Decodable Account. Like this:

• You'll create and configure an IAM Role with two Policies:
  • A Trust Policy allowing access from Decodable's AWS account — but only with an ExternalId matching your (unique) Decodable account name.
  • A Permissions Policy with the needed permissions on your stream.
• You'll provide us the ARN of this Role via your Decodable Kinesis connection's aws.credentials.role.arn property.
• Our servers will assume that Role using an ExternalId value matching only your Decodable Account name — never any other. We'll use that to talk to your bucket.

Note that the values here are not treated as secret (by us, AWS, or you): not ExternalId (your account name), not the Role ARN, not the bucket name.

Specifically, your IAM Role (per-roleArn) must:

• have an AssumeRole Trust Policy that:
  • names Decodable's AWS account ID (671293015970) as Principal.
  • has a Condition requiring sts:ExternalId to equal your Decodable Account name.
• have a Permissions Policy allowing needed operations on the stream (not Role) ARN

For example

Here's an example IAM Trust Policy. Replace my-decodable-account. Note that 671293015970 is Decodable's AWS account ID and must match exactly.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::671293015970:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "my-decodable-account"
        }
      }
    }
  ]
}

Note: To allow several Decodable Accounts (say, in different AWS Regions) to write to the same bucket, use an array of Account names for the ExternalId value:

{ "sts:ExternalId": ["my-acct-1", "my-acct-2"] }

Here's an example IAM Permissions Policy to allow full access to your Kinesis streams..

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kinesis:*",
      "Resource": "*"
    }
  ]
}

Note that you can restrict down the permissions for specific Kinesis streams by specifying a Kinesis ARN instead of the wildcard "*", as well as listing out specific Action permissions.

Further reading — from AWS

For full discussion from AWS of the security problem this solves, and its AWS-recommended solution using ExternalId,
we recommend reading:

AWS Identity and Access Management • The confused deputy problem.