Resource definitions

If you are new to declarative resource management in Decodable see the overview.

This is the reference for the apply input and query output YAML format.

There are four resource types in Decodable:

  • Connection

  • Stream

  • Pipeline

  • Secret

Each resource definition takes the form of a YAML document, which begins with a line of three hyphens ---. A YAML file is a series of YAML documents. If you’re new to YAML, you may find this excellent tutorial that covers the basics and advanced concepts useful.

For a complete example with a set of resource definitions that work together, see here.

Specification versions (spec_version)

All resource definitions support the v1 specification version. Connections and pipelines also support v2.

v2 adds support for execution details of the resource, such as state and task size.

Top-level structure

Each resource definition YAML document has these fields:

---
kind: <resource_type>
metadata:
  name: <name_your_resource>
  description: <resource_description>
spec_version: <v1|v2>
spec:
  <resource_specifications>
Field Required? Description

kind

Required

The kind of resource. One of: secret, connection, pipeline, stream.

metadata

Required

Values that identify or describe the resource, including name and description text strings. The name value is mapped to an ID when the YAML is applied. The description field is optional.

spec_version

Required

Determines spec format and processing logic for this kind.

  • Must be v1 for streams and secrets.

  • Must be v1 or v2 for connections and pipelines.

spec

Required, except with kind: secret

The resource’s specifications. The format of the spec object is specific to each resource kind and spec_version combination. It contains nested fields specific to that resource.

Status

The query output format extends the above resource definition structure with a status section. The status contains non-writable fields managed by Decodable for the resource, and is ignored by apply.

An example status section looks like this:

status:
  version: 1
  is_latest: true
  active_version: 1
  latest_version: 1
  create_time: 2024-07-23T20:57:58.760+00:00
  last_activated_time: 2024-07-23T21:45:47.276+00:00
  execution:
    state: RUNNING

Connection

A template for a connection definition:

---
kind: connection
metadata:
  name: <name_your_connection>
  description: <description>
spec_version: v2
spec:
  connector: <connector_name>
  type: <type of connector>
  properties:
    <property_1>: <value>
    <property_2>: <value>
  stream_name: <stream>
  schema_v2:
    fields:
      - kind: physical
        name: <field name>
        type: <data type>
      - kind: computed
        name: <field name>
        expression: <data type>
      - kind: metadata
        name: <field name>
        type: <data type>
        key: <per-connector key>
  execution: # v2 spec only
    active: <true|false>
    task_size: <task size>
    task_count: <number of tasks>
Field Description

v1 and v2 spec

spec.connector

The name of the connector. For example: mysql-cdc or s3-v2.

spec.type

The type of connector. Enter source if your connector receives data or sink if your connector sends data.

spec.properties

Each connector has its own set of properties that you can specify. To determine the property names and their valid values for a specific connector, refer to the corresponding documentation page for that connector.

Important: Secret properties, such as passwords, use the name of a secret resource, not the actual password or secret plaintext value.

The following are only valid for connectors that support a single stream

spec.stream_name

The name of the stream that you want the connection to send or receive data from.

spec.schema_v2

The schema of the connection. This must be compatible with the schema of the stream that it’s connected to.

spec.schema_v2.fields[*].kind

One of: physical, computed, metadata.

For more information on how to use each field kind, see here.

The following are only valid for connectors that support a multiple streams

spec.stream_mappings

One or more mappings between Decodable streams and external resources in the system that the connection reads from or writes to.

Consult the connector documentation for the specific resource specifiers available.

For example:

spec:
  […]
  stream_mappings:
      - stream_name: epos_orders
        external_resource_specifier:
          database-name: camelot
          table-name: orders
      - stream_name: epos_products
        external_resource_specifier:
          database-name: merlin
          table-name: products_v2
      - stream_name: epos_products
        external_resource_specifier:
          database-name: excalibur
          table-name: products

Stream:external resource mappings may be 1:1, 1:N (in the case of a source), or N:1 (in the case of a sink).

That is, many external resources can be mapped to a single stream in a source, and many streams can be mapped to a single external resource in a sink.

v2 spec

spec.execution.active

Whether the connection is active or not.

Valid values: true, false

Default: false

spec.execution.task_size

The task size for the connection.

Valid values: S, M, L

Default: M

spec.execution.task_count

The number of tasks to run for the connection.

Default: 1

spec.execution.initial_start_position

When starting the connection for the first time, or after a reset-state operation, the start position in the source streams from which to read.

Valid values: earliest, latest

Default: latest

Stream

A template for a stream definition:

---
kind: stream
metadata:
  name: <name_your_stream>
  description: <description>
spec_version: v1
spec:
  schema_v2:
    fields:
      - kind: physical
        name: <field name>
        type: <data type>
      - kind: computed
        name: <field name>
        expression: <data type>
    constraints:  # optional
      primary_key:
        - <field name>
    watermarks:  # optional
      - name: <name>
        expression: <watermark expression>
Field Required? Description

spec.schema_v2

Required

The schema of the stream. This must be compatible with the schema of any connection that it’s connected to, and any pipeline that uses it.

spec.schema_v2.fields[*].kind

Required

One of: physical, computed.

For more information on how to use each field kind, see here.

spec.schema_v2.constraints.primary_key

Optional

Specify one or more fields to use as a primary key or partition key.

You must (only) specify a primary key if you are sending Change Data Capture (CDC) records. For more information on partition keys or primary keys, see the table in How to use the schema view.

spec.schema_v2.watermarks

Optional

A field that can be used to track the progression of event time in a pipeline.

Pipeline

SQL Pipeline

A template for a SQL Pipeline definition:

---
kind: pipeline
metadata:
  name: <name_your_pipeline>
  description: <description>
spec_version: v2
spec:
  sql: |
    INSERT INTO stream_out
    SELECT ...
    FROM stream_in
  execution: # v2 spec only
    active: <true|false>
    task_size: <task size>
    task_count: <number of tasks>
    initial_start_positions:
      stream_1: earliest
      stream_2: latest

Field

Description

v1 and v2 spec

spec.sql

A SQL statement in the form:

INSERT INTO <output-stream>
SELECT ... FROM <input-stream>

For example:

INSERT INTO stream_out
SELECT LOWER(col1) FROM stream_in

v2 spec only

spec.execution.active

Whether the pipeline is active or not. Boolean

Default: false

spec.execution.task_size

The task size for the pipeline.

Valid values: S, M, L

Default: M

spec.execution.task_count

The number of tasks to run for the pipeline.

Default: 1

spec.execution.initial_start_positions

For each stream, the start position from which to read when starting the pipeline for the first time, or after a reset-state operation.

Specify the stream name and its starting position.

Valid values: earliest, latest

Default: latest

For example:

spec:
  execution:
    initial_start_positions:
      stream_1: earliest
      stream_2: latest
Can’t be used with initial_snapshot_id.

spec.execution.initial_snapshot_id

When starting the pipeline for the first time, or after a reset-state operation, the snapshot from which to start.

For example:

spec:
  execution:
    initial_snapshot_id: 4e335631
Can’t be used with initial_start_positions.

Custom Pipeline

A template for a Custom Pipeline definition. All referenced files are assumed to be available in the same location as the YAML doc:

---
kind: pipeline
metadata:
  name: <name_your_pipeline>
  description: <description>
spec_version: v2
spec:
  type: JAVA
  job_file_path: pipeline.jar
  config_file_paths:
    - config.yaml
    - example.data
  properties:
    secrets:
      - <secret_name>
    flink_version: 1.18-java11
    additional_metrics:
      - some.flink.metric
  execution: # v2 spec only
    active: <true|false>
    task_size: <task size>
    task_count: <number of tasks>
    initial_snapshot_id: <snapshot id>

Field

Description

Required?

v1 and v2 spec

spec.type

Job file type. Either JAVA or PYTHON.

Required

spec.job_file_path

Path to job file (a JAR for JAVA, a ZIP for PYTHON) in local file system. CLI will upload this file as needed. Not available if invoking the API directly.

Either this or job_file_sha is required

spec.job_file_sha

SHA-512 of contents of job file. Must match an already uploaded job file.

Either this or job_file_path is required

spec.job_arguments

Argument string input to the Custom Pipeline program.

Optional

spec.entry_class

Entry class of the Custom Pipeline. If not provided, the entry class must be specified in the file META-INF/MANIFEST.MF in the pipeline’s JAR file, using the Main-Class property key.

Optional

spec.config_file_paths

List of strings, each a path to a configuration file in local file system. CLI will upload these files as needed. Not available if invoking the API directly.

Optional

spec.config_files

List of objects (as follows).

Optional

spec.config_files[*].name

Name of configuration file as exposed to the Custom Pipeline program.

Optional

spec.config_files[*].sha

SHA-512 of contents of configuration file. Must match an already uploaded configuration file.

Required

spec.properties

Object (as follows).

Required

spec.properties.secrets

List of strings, each a Secret name (or ID).

Optional

spec.properties.flink_version

A specific Decodable-supported Flink version (string).

Required

spec.properties.additional_metrics

List of strings, each a Flink metric to expose in the _metrics stream for this Custom Pipeline.

Optional

v2 spec only

spec.execution.active

Whether the pipeline is active or not. Boolean

Default: false

Optional

spec.execution.task_size

The task size for the pipeline.

Valid values: S, M, L

Default: M

Optional

spec.execution.task_count

The number of tasks to run for the pipeline.

Default: 1

Optional

spec.execution.initial_start_positions

For each stream, the start position from which to read when starting the pipeline for the first time, or after a reset-state operation.

Specify the stream name and its starting position.

Valid values: earliest, latest

Default: latest

Specify the stream name and its starting position. For example:

spec:
  execution:
    initial_start_positions:
      stream_1: earliest
      stream_2: latest
Can’t be used with initial_snapshot_id

Optional

spec.execution.initial_snapshot_id

When starting the pipeline for the first time, or after a reset-state operation, the snapshot from which to start.

For example:

spec:
  execution:
    initial_snapshot_id: 4e335631
Can’t be used with initial_start_positions.

Optional

Secret

A template for a secret definition:

---
kind: secret
metadata:
  name: <name_your_secret>
  description: <description>
spec_version: v1
spec:
  # Use _one_ of the following value_* fields.
  value_env_var: MY_PASSWORD
  value_file: ./secret.txt
  value_literal: "don't-tell-123"

A Decodable Secret resource stores a plaintext value (such as a password) securely, so that it may be used by Connections or Custom Pipelines. A Connection references a Secret by name from a (secret-type) property value.

The apply CLI command can set this plaintext value from an environment variable, a file, or a literal value, via special spec fields.

These fields can only be used to write the plaintext value: the query command output for a Secret will not include them. The fields aren’t considered part of the Secret resource. Instead they act as directives to the CLI apply command to write the plaintext value to the secure Secret storage in your Decodable account.
Field Description

spec.value_env_var

The name of an environment variable containing the plaintext value.

This is recommended for a simple plaintext value string such as a password.

Note that not all binary values can be stored in an environment variable, and that some shell-based techniques can strip newlines when setting an environment variable.

The apply command will error if no environment variable by this name is set.

spec.value_file

A path (relative to the YAML file’s directory) to a local file containing the plaintext value.

This is recommended for a multiline string (such as a private key) or binary plaintext value.

The file contents are used verbatim. Note that this includes any trailing newlines.

The apply command will error if no such file can be found and read.

spec.value_literal

A string with the literal plaintext value.

Be cautious storing any secret value this way. Care must be taken to ensure that any YAML file that uses this field isn’t committed to source control, emailed, sent over Slack, logged, etc.

This field may be appropriate for private ad-hoc use, and for use with generated YAML. The other two fields are recommended for most CI/CD use.

For API users: These fields are specially handled by the CLI apply command; the underlying /apply API doesn’t directly support them.

Secret definition without plaintext value

When no spec.value_* field is present—​as in query command output—​no plaintext value is written for the Secret resource on apply.

In this case there is no spec for the Secret definition, but spec_version: v1 is still required.

For a Secret created this way, the plaintext value may be manually set after apply, as a separate step. This can be done via the Decodable UI or CLI (decodable secret write <id> --value '<value>'). It need only be done once per Secret. Alternately, a spec.value_* field may be added and apply repeated.

If the Secret resource already exists and no spec.value_* field is provided then the existing plaintext value will remain unchanged. This may be appropriate for certain use cases.