AWS Kinesis
Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.
Getting Started
Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kinesis connectors can be used in either role, performing as either a source
or a sink
.
Configure As A Source
To create and configure a connector for Kinesis, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection
, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.
-
Select a connector type, either
source
orsink
, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system. -
Provide the name of your Kinesis stream.
-
Provide the AWS region of your Kinesis stream.
-
Specify the AWS ARN of the IAM role. For example,
arn:aws:iam::111222333444:role/decodable-kinesis-access
. -
The value format will default to
JSON
, since that is the only option for Kinesis connectors.
For more detailed information about Kinesis, see the Kinesis Getting Started
guide and related documentation.Note: The AWS Kinesis connector might require task size L tasks when consuming from streams with many shards (e.g. streams that have been created with high data volumes). See Manage task count and sizes.
Configure As A Sink
To create a Kinesis connector for use as a sink
, all the configuration steps above apply.
Reference
Connector name | kinesis |
Type | source , sink |
Delivery guarantee | at least once |
The Kinesis connector streams data into or out of a Kinesis stream in your AWS account. To use it, configure an AWS IAM Role as described below, with specific permissions to access the stream.
Properties
If you are using the Decodable CLI to create or edit a connection to a Kinesis stream, then use the following table as a reference for what properties are required and supported.
Property | Disposition | Description |
---|---|---|
aws.region | required | Region of the Kinesis stream.br/>Example: us-west-2 |
aws.credentials.role.arn | required | AWS ARN of the IAM Role configured as described below. Example: arn:aws:iam::111222333444:role/decodable-kinesis-access . |
stream | required | Name of the Kinesis stream. |
format | required | Must be json . |
IAM Role, Permissions, and Security
To be secure, you, AWS, and Decodable work together to ensure only Kinesis connections in your Decodable Account can access your Kinesis streams.
How?
AWS IAM provides a special mechanism — called ExternalId
— that you and Decodable will use as described here, which ensures access from Decodable to your stream happens only for your Decodable Account. Like this:
- You'll create and configure an IAM Role with two Policies:
- A Trust Policy allowing access from Decodable's AWS account — but only with an
ExternalId
matching your (unique) Decodable account name. - A Permissions Policy with the needed permissions on your stream.
- A Trust Policy allowing access from Decodable's AWS account — but only with an
- You'll provide us the ARN of this Role via your Decodable Kinesis connection's
aws.credentials.role.arn
property. - Our servers will assume that Role using an
ExternalId
value matching only your Decodable Account name — never any other. We'll use that to talk to your bucket.
Note that the values here are not treated as secret (by us, AWS, or you): not ExternalId
(your account name), not the Role ARN, not the bucket name.
Specifically, your IAM Role (per-roleArn
) must:
- have an
AssumeRole
Trust Policy that:- names Decodable's AWS account ID (
671293015970
) asPrincipal
. - has a
Condition
requiringsts:ExternalId
to equal your Decodable Account name.
- names Decodable's AWS account ID (
- have a Permissions Policy allowing needed operations on the stream (not Role) ARN
For example
Here's an example IAM Trust Policy. Replace my-decodable-account
. Note that 671293015970
is Decodable's AWS account ID and must match exactly.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::671293015970:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "my-decodable-account"
}
}
}
]
}
Note: To allow several Decodable Accounts (say, in different AWS Regions) to write to the same bucket, use an array of Account names for the
ExternalId
value:
{ "sts:ExternalId": ["my-acct-1", "my-acct-2"] }
Here's an example IAM Permissions Policy to allow full access to your Kinesis streams..
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "kinesis:*",
"Resource": "*"
}
]
}
Note that you can restrict down the permissions for specific Kinesis streams by specifying a Kinesis ARN instead of the wildcard "*"
, as well as listing out specific Action permissions.
Further reading — from AWS
For full discussion from AWS of the security problem this solves, and its AWS-recommended solution using ExternalId
,
we recommend reading:
AWS Identity and Access Management • The confused deputy problem.
Updated about 2 months ago