Apache Iceberg sink connector Use the Apache Iceberg connector to send data to an Apache Iceberg table. Features Delivery guarantee Exactly once Compatibility This connector currently supports the following catalog and data warehouse options: Catalog: AWS Glue Data Warehouse: Amazon S3 Prerequisites Access to your AWS resources Decodable interacts with resources in AWS on your behalf. To do this you need an IAM role configured with a trust policy that allows access from Decodable’s AWS account, and a permission policy as detailed below. For more details on how this works, how to configure the trust policy, and example steps to follow see here. To use this connector you must associate a permissions policy with the IAM role. This policy must have the following permissions: Read/Write access to the S3 bucket path to which you’re writing data. s3:PutObject s3:GetObject s3:DeleteObject If you want to send data directly at the root level of the bucket, then leave the path blank with the trailing /* included. List access on the bucket to which you’re writing data s3:ListBucket Read and write permissions for AWS Glue. glue:CreateDatabase glue:GetDatabase glue:CreateTable glue:UpdateTable glue:GetTable Sample Permission Policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:CreateDatabase", "glue:GetDatabase", "glue:CreateTable", "glue:UpdateTable", "glue:GetTable" ], "Resource": [ "arn:aws:glue:<region>:<AWS account id>:catalog", "arn:aws:glue:<region>:<AWS account id>:database/<catalog-database>", "arn:aws:glue:<region>:<AWS account id>:table/<catalog-database>/<catalog-table>" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<s3 warehouse path>/*" ] } ] } In this example, replace: <region> with the AWS region of the Glue catalog. <AWS account id> with your AWS account ID. <s3 warehouse path>, <catalog-database>, and <catalog-table> with appropriate values for your Iceberg tables. Steps If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is iceberg. From the Connections page, select the Iceberg Connector and complete the following fields. UI Field Property Name Description Connection Type N/A Select Sink to use this connector to send data into the database provided. Warehouse path warehouse The file path to the S3 bucket or folder that you want to send data to. For example: s3://bucket/folder. Database name catalog-database The name of the database in your Iceberg catalog. This is the name that you added permissions for as part of the prerequisites. If a database with this name doesn’t exist, Decodable creates it. Table name catalog-table The name of the table in your Iceberg catalog. This is the name that you added permissions for as part of the prerequisites. If a table with this name doesn’t exist, Decodable creates it. Catalog type catalog-type The catalog responsible for managing the metadata associated with Iceberg tables. Currently, only AWS Glue is supported. If you are using the Decodable CLI to create this connection, enter glue for this value. IAM Role ARN role-arn The AWS ARN of the IAM role. For example, arn:aws:iam::111222333444:role/decodable-s3-access. AWS Region region The AWS region of the AWS Glue catalog. Format format The format for data in Amazon S3. The following formats are supported: parquet avro orc Defaults to parquet. Table Format Version format-version The Iceberg table specification version used for output files: 1 2 (default) See the note below regarding versions. Select which stream contains the records that you’d like to send to an Iceberg table. Then, select Next. Give the newly created connection a Name and Description and select Save. Upon starting this connection, you can use it to send data to your Iceberg table. If you are sending data from a change stream to an Iceberg table, then upsert mode is used. Otherwise, append mode is used. Iceberg table spec version This connector defaults to using version 2 of the Iceberg table specification. Support for version 1 is provided for backwards compatibility. Note that version 1 has limited support for change streams. It doesn’t support consuming deletion records and will fail at runtime if it encounters one. In addition, updates cause entire files to be rewritten, increasing cost and decreasing performance. We therefore recommend using version 2 when working with change streams. If necessary, use a Pipeline to convert the stream into an append stream with the TO_APPEND function.