Resource definitions If you are new to declarative configuration in Decodable see the overview. This is the reference for the apply input and query output YAML format. There are four resource types in Decodable: Connection Stream Pipeline Secret Each resource definition takes the form of a YAML document, which begins with a line of three hyphens ---. A YAML file is a series of YAML documents. If you’re new to YAML, you may find this excellent tutorial that covers the basics and advanced concepts useful. For a complete example with a set of resource definitions that work together, see here. Specification versions (spec_version) All resource definitions support the v1 specification version. Connections and pipelines also support v2. v2 adds support for execution details of the resource, such as state and task size. Top-level structure Each resource definition YAML document has these fields: --- kind: <resource_type> metadata: name: <name_your_resource> description: <resource_description> tags: <tag_key_1>: <tag_value_1> spec_version: <v1|v2> spec: <resource_specifications> Field Required? Description kind Required The kind of resource. One of: secret, connection, pipeline, stream. metadata Required Values that identify or describe the resource, including name, description, and tags. The name value is mapped to an ID when the YAML is applied. The description field is optional and provides additional context about the resource. The tags section is optional and consists of key/value pairs that help categorize the resource. Each tag key must be unique and is associated with an optional value. spec_version Required Determines spec format and processing logic for this kind. Must be v1 for streams and secrets. Must be v1 or v2 for connections and pipelines. spec Required, except with kind: secret The resource’s specifications. The format of the spec object is specific to each resource kind and spec_version combination. It contains nested fields specific to that resource. Status The query output format extends the above resource definition structure with a status section. The status contains non-writable fields managed by Decodable for the resource, and is ignored by apply. An example status section looks like this: status: version: 1 is_latest: true active_version: 1 latest_version: 1 create_time: 2024-07-23T20:57:58.760+00:00 last_activated_time: 2024-07-23T21:45:47.276+00:00 execution: state: RUNNING actual_system_flink_configuration: operator-chaining: "true" system_flink_configuration: operator-chaining: "true" Field Description v1 and v2 spec status.actual_system_flink_configuration This field is only present when the job is RUNNING and shows the current Flink configurations applied to the running job. status.system_flink_configuration The latest Flink configurations defined for this job. When this differs from actual_system_flink_configuration, restart the job to apply the latest configurations. Connection A template for a connection definition: --- kind: connection metadata: name: <name_your_connection> description: <description> tags: <tag_key_1>: <tag_value_1> spec_version: v2 spec: connector: <connector_name> type: <type of connector> properties: <property_1>: <value> <property_2>: <value> stream_name: <stream> schema_v2: fields: - kind: physical name: <field name> type: <data type> - kind: computed name: <field name> expression: <data type> - kind: metadata name: <field name> type: <data type> key: <per-connector key> execution: # v2 spec only active: <true|false> task_size: <task size> task_count: <number of tasks> Field Description v1 and v2 spec spec.connector The name of the connector. For example: mysql-cdc or s3-v2. spec.type The type of connector. Enter source if your connector receives data or sink if your connector sends data. spec.properties Each connector has its own set of properties that you can specify. To determine the property names and their valid values for a specific connector, refer to the corresponding documentation page for that connector. Important: Secret properties, such as passwords, use the name of a secret resource, not the actual password or secret plaintext value. The following are only valid for connectors that support a single stream spec.stream_name The name of the stream that you want the connection to send or receive data from. spec.schema_v2 The schema of the connection. This must be compatible with the schema of the stream that it’s connected to. spec.schema_v2.fields[*].kind One of: physical, computed, metadata. For more information on how to use each field kind, see here. The following are only valid for connectors that support a multiple streams spec.stream_mappings One or more mappings between Decodable streams and external resources in the system that the connection reads from or writes to. Consult the connector documentation for the specific resource specifiers available. For example: spec: […] stream_mappings: - stream_name: epos_orders external_resource_specifier: database-name: camelot table-name: orders - stream_name: epos_products external_resource_specifier: database-name: merlin table-name: products_v2 - stream_name: epos_products external_resource_specifier: database-name: excalibur table-name: products Stream:external resource mappings may be 1:1, 1:N (in the case of a source), or N:1 (in the case of a sink). That is, many external resources can be mapped to a single stream in a source, and many streams can be mapped to a single external resource in a sink. v2 spec spec.execution.active Whether the connection is active or not. Valid values: true, false Default: false spec.execution.task_size The task size for the connection. Valid values: S, M, L, LM2, LM4 Default: M spec.execution.task_count The number of tasks to run for the connection. Default: 1 spec.execution.initial_start_position When starting the connection for the first time, or after a reset-state operation, the start position in the source streams from which to read. Valid values: earliest, latest Default: latest Stream A template for a stream definition: --- kind: stream metadata: name: <name_your_stream> description: <description> tags: <tag_key_1>: <tag_value_1> spec_version: v1 spec: schema_v2: fields: - kind: physical name: <field name> type: <data type> - kind: computed name: <field name> expression: <data type> constraints: # optional primary_key: - <field name> watermarks: # optional - name: <name> expression: <watermark expression> Field Required? Description spec.type Optional The stream type, either "append" or "change." If omitted, the stream type is inferred by the presence of a primary key. To use a primary key with an append stream you must specify the stream type explicitly. With no type specified, a primary key implies a change stream. spec.partition_key Optional Specify one or more fields to use as a partition key. Deprecated: It’s recommended to set the stream type to append and use the spec.schema_v2.primary_key setting instead of setting the spec.partition_key directly. spec.schema_v2 Required The schema of the stream. This must be compatible with the schema of any connection that it’s connected to, and any pipeline that uses it. spec.schema_v2.fields[*].kind Required One of: physical, computed. For more information on how to use each field kind, see here. spec.schema_v2.constraints.primary_key Optional Specify one or more fields to use as a primary key or partition key. You must (only) specify a primary key if you are sending Change Data Capture (CDC) records. For more information on partition keys or primary keys, see the table in How to use the schema view. spec.schema_v2.watermarks Optional A field that can be used to track the progression of event time in a pipeline. Pipeline SQL Pipeline A template for a SQL Pipeline definition: --- kind: pipeline metadata: name: <name_your_pipeline> description: <description> tags: <tag_key_1>: <tag_value_1> spec_version: v2 spec: sql: | INSERT INTO stream_out SELECT ... FROM stream_in execution: # v2 spec only active: <true|false> task_size: <task size> task_count: <number of tasks> initial_start_positions: stream_1: earliest stream_2: latest Field Description v1 and v2 spec spec.sql A SQL statement in the form: INSERT INTO <output-stream> SELECT ... FROM <input-stream> For example: INSERT INTO stream_out SELECT LOWER(col1) FROM stream_in v2 spec only spec.execution.active Whether the pipeline is active or not. Boolean Default: false spec.execution.task_size The task size for the pipeline. Valid values: S, M, L, LM2, LM4 Default: M spec.execution.task_count The number of tasks to run for the pipeline. Default: 1 spec.execution.initial_start_positions For each stream, the start position from which to read when starting the pipeline for the first time, or after a reset-state operation. Specify the stream name and its starting position. Valid values: earliest, latest Default: latest For example: spec: execution: initial_start_positions: stream_1: earliest stream_2: latest Can’t be used with initial_snapshot_id. spec.execution.initial_snapshot_id When starting the pipeline for the first time, or after a reset-state operation, the snapshot from which to start. For example: spec: execution: initial_snapshot_id: 4e335631 Can’t be used with initial_start_positions. Custom Pipeline A template for a Custom Pipeline definition. All referenced files are assumed to be available in the same location as the YAML doc: --- kind: pipeline metadata: name: <name_your_pipeline> description: <description> tags: <tag_key_1>: <tag_value_1> spec_version: v2 spec: type: JAVA job_file_path: pipeline.jar config_file_paths: - config.yaml - example.data properties: secrets: - <secret_name> flink_version: 1.18-java11 additional_metrics: - some.flink.metric execution: # v2 spec only active: <true|false> task_size: <task size> task_count: <number of tasks> initial_snapshot_id: <snapshot id> Field Required? Description v1 and v2 spec spec.type Required Job file type. Either JAVA or PYTHON. spec.job_file_path Either this or job_file_sha is required Path to job file (a JAR for JAVA, a ZIP for PYTHON) in local file system. CLI will upload this file as needed. Not available if invoking the API directly. spec.job_file_sha Either this or job_file_path is required SHA-512 of contents of job file. Must match an already uploaded job file. spec.job_arguments Optional Argument string input to the Custom Pipeline program. spec.entry_class Optional Entry class of the Custom Pipeline. If not provided, the entry class must be specified in the file META-INF/MANIFEST.MF in the pipeline’s JAR file, using the Main-Class property key. spec.config_file_paths Optional List of strings, each a path to a configuration file in local file system. CLI will upload these files as needed. Not available if invoking the API directly. spec.config_files Optional List of objects (as follows). spec.config_files[*].name Optional Name of configuration file as exposed to the Custom Pipeline program. spec.config_files[*].sha Required SHA-512 of contents of configuration file. Must match an already uploaded configuration file. spec.properties Required Object (as follows). spec.properties.secrets Optional List of strings, each a Secret name (or ID). spec.properties.flink_version Required A specific Decodable-supported Flink version (string). spec.properties.additional_metrics Optional List of strings, each a Flink metric to expose in the _metrics stream for this Custom Pipeline. v2 spec only spec.execution.active Optional Whether the pipeline is active or not. Boolean Default: false spec.execution.task_size Optional The task size for the pipeline. Valid values: S, M, L, LM2, LM4 Default: M spec.execution.task_count Optional The number of tasks to run for the pipeline. Default: 1 spec.execution.initial_start_positions Optional For each stream, the start position from which to read when starting the pipeline for the first time, or after a reset-state operation. Specify the stream name and its starting position. Valid values: earliest, latest Default: latest Specify the stream name and its starting position. For example: spec: execution: initial_start_positions: stream_1: earliest stream_2: latest Can’t be used with initial_snapshot_id spec.execution.initial_snapshot_id Optional When starting the pipeline for the first time, or after a reset-state operation, the snapshot from which to start. For example: spec: execution: initial_snapshot_id: 4e335631 Can’t be used with initial_start_positions. Secret A template for a secret definition: --- kind: secret metadata: name: <name_your_secret> description: <description> tags: <tag_key_1>: <tag_value_1> spec_version: v1 spec: # Use _one_ of the following value_* fields. value_env_var: MY_PASSWORD value_file: ./secret.txt value_literal: "don't-tell-123" A Decodable Secret resource stores a plaintext value (such as a password) securely, so that it may be used by Connections or Custom Pipelines. A Connection references a Secret by name from a (secret-type) property value. The apply CLI command can set this plaintext value from an environment variable, a file, or a literal value, via special spec fields. These fields can only be used to write the plaintext value: the query command output for a Secret will not include them. The fields aren’t considered part of the Secret resource. Instead they act as directives to the CLI apply command to write the plaintext value to the secure Secret storage in your Decodable account. Field Description spec.value_env_var The name of an environment variable containing the plaintext value. This is recommended for a simple plaintext value string such as a password. Note that not all binary values can be stored in an environment variable, and that some shell-based techniques can strip newlines when setting an environment variable. The apply command will error if no environment variable by this name is set. spec.value_file A path (relative to the YAML file’s directory) to a local file containing the plaintext value. This is recommended for a multiline string (such as a private key) or binary plaintext value. The file contents are used verbatim. Note that this includes any trailing newlines. The apply command will error if no such file can be found and read. spec.value_literal A string with the literal plaintext value. Be cautious storing any secret value this way. Care must be taken to ensure that any YAML file that uses this field isn’t committed to source control, emailed, sent over Slack, logged, etc. This field may be appropriate for private ad-hoc use, and for use with generated YAML. The other two fields are recommended for most CI/CD use. For API users: These fields are specially handled by the CLI apply command; the underlying /apply API doesn’t directly support them. Secret definition without plaintext value When no spec.value_* field is present—as in query command output—no plaintext value is written for the Secret resource on apply. In this case there is no spec for the Secret definition, but spec_version: v1 is still required. For a Secret created this way, the plaintext value may be manually set after apply, as a separate step. This can be done via the Decodable UI or CLI (decodable secret write <id> --value '<value>'). It need only be done once per Secret. Alternately, a spec.value_* field may be added and apply repeated. If the Secret resource already exists and no spec.value_* field is provided then the existing plaintext value will remain unchanged. This may be appropriate for certain use cases.