Resource definitions
If you are new to declarative resource management in Decodable see the overview. |
There are four resource types in Decodable:
-
Connection
-
Stream
-
Pipeline
-
Secret
Each resource definition takes the form of a YAML document, which begins with a line of three hyphens ---
.
A YAML file is a series of YAML documents.
If you’re new to YAML, you may find this excellent tutorial that covers the basics and advanced concepts useful.
For a complete example with a set of resource definitions that work together, see here.
Specification versions (spec_version
)
All resource definitions support the v1
specification version. Connections and pipelines also support v2
.
v2
adds support for execution details of the resource, such as state and task size.
Top-level structure
Each resource definition YAML document has these fields:
---
kind: <resource_type>
metadata:
name: <name_your_resource>
description: <resource_description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: <v1|v2>
spec:
<resource_specifications>
Field | Required? | Description |
---|---|---|
|
Required |
The kind of resource. One of: |
|
Required |
Values that identify or describe the resource, including
|
|
Required |
Determines
|
|
Required, except with |
The resource’s specifications.
The format of the |
Status
The query
output format extends the above resource definition structure with a status
section.
The status
contains non-writable fields managed by Decodable for the resource, and is ignored by apply
.
An example status
section looks like this:
status:
version: 1
is_latest: true
active_version: 1
latest_version: 1
create_time: 2024-07-23T20:57:58.760+00:00
last_activated_time: 2024-07-23T21:45:47.276+00:00
execution:
state: RUNNING
Connection
A template for a connection definition:
---
kind: connection
metadata:
name: <name_your_connection>
description: <description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: v2
spec:
connector: <connector_name>
type: <type of connector>
properties:
<property_1>: <value>
<property_2>: <value>
stream_name: <stream>
schema_v2:
fields:
- kind: physical
name: <field name>
type: <data type>
- kind: computed
name: <field name>
expression: <data type>
- kind: metadata
name: <field name>
type: <data type>
key: <per-connector key>
execution: # v2 spec only
active: <true|false>
task_size: <task size>
task_count: <number of tasks>
Field | Description | ||
---|---|---|---|
|
|||
|
The name of the connector. For example: |
||
|
The type of connector. Enter |
||
|
Each connector has its own set of properties that you can specify. To determine the property names and their valid values for a specific connector, refer to the corresponding documentation page for that connector. Important: Secret properties, such as passwords, use the name of a secret resource, not the actual password or secret plaintext value. |
||
The following are only valid for connectors that support a single stream |
|||
|
The name of the stream that you want the connection to send or receive data from. |
||
|
The schema of the connection. This must be compatible with the schema of the stream that it’s connected to. |
||
|
One of: For more information on how to use each field |
||
The following are only valid for connectors that support a multiple streams |
|||
|
One or more mappings between Decodable streams and external resources in the system that the connection reads from or writes to. Consult the connector documentation for the specific resource specifiers available. For example:
yaml
|
||
|
Whether the connection is active or not. Valid values: Default: |
||
|
The task size for the connection. Valid values: Default: |
||
|
The number of tasks to run for the connection. Default: |
||
|
When starting the connection for the first time, or after a Valid values: Default: |
Stream
A template for a stream definition:
---
kind: stream
metadata:
name: <name_your_stream>
description: <description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: v1
spec:
schema_v2:
fields:
- kind: physical
name: <field name>
type: <data type>
- kind: computed
name: <field name>
expression: <data type>
constraints: # optional
primary_key:
- <field name>
watermarks: # optional
- name: <name>
expression: <watermark expression>
Field | Required? | Description |
---|---|---|
|
Required |
The schema of the stream. This must be compatible with the schema of any connection that it’s connected to, and any pipeline that uses it. |
|
Required |
One of: For more information on how to use each field |
|
Optional |
Specify one or more fields to use as a primary key or partition key. You must (only) specify a primary key if you are sending Change Data Capture (CDC) records. For more information on partition keys or primary keys, see the table in How to use the schema view. |
|
Optional |
A field that can be used to track the progression of event time in a pipeline. |
Pipeline
SQL Pipeline
A template for a SQL Pipeline definition:
---
kind: pipeline
metadata:
name: <name_your_pipeline>
description: <description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: v2
spec:
sql: |
INSERT INTO stream_out
SELECT ...
FROM stream_in
execution: # v2 spec only
active: <true|false>
task_size: <task size>
task_count: <number of tasks>
initial_start_positions:
stream_1: earliest
stream_2: latest
Field |
Description |
||
|
|||
|
A SQL statement in the form:
sql For example:
sql |
||
|
Whether the pipeline is active or not. Boolean Default: |
||
|
The task size for the pipeline. Valid values: Default: |
||
|
The number of tasks to run for the pipeline. Default: |
||
|
For each stream, the start position from which to read when starting the pipeline for the first time, or after a Specify the stream name and its starting position. Valid values: Default: For example:
yaml
|
||
|
When starting the pipeline for the first time, or after a For example:
yaml
|
Custom Pipeline
A template for a Custom Pipeline definition. All referenced files are assumed to be available in the same location as the YAML doc:
---
kind: pipeline
metadata:
name: <name_your_pipeline>
description: <description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: v2
spec:
type: JAVA
job_file_path: pipeline.jar
config_file_paths:
- config.yaml
- example.data
properties:
secrets:
- <secret_name>
flink_version: 1.18-java11
additional_metrics:
- some.flink.metric
execution: # v2 spec only
active: <true|false>
task_size: <task size>
task_count: <number of tasks>
initial_snapshot_id: <snapshot id>
Field |
Description |
Required? |
||
|
||||
|
Job file type. Either |
Required |
||
|
Path to job file (a JAR for |
Either this or |
||
|
SHA-512 of contents of job file. Must match an already uploaded job file. |
Either this or |
||
|
Argument string input to the Custom Pipeline program. |
Optional |
||
|
Entry class of the Custom Pipeline. If not provided, the entry class must be specified in the file |
Optional |
||
|
List of strings, each a path to a configuration file in local file system. CLI will upload these files as needed. Not available if invoking the API directly. |
Optional |
||
|
List of objects (as follows). |
Optional |
||
|
Name of configuration file as exposed to the Custom Pipeline program. |
Optional |
||
|
SHA-512 of contents of configuration file. Must match an already uploaded configuration file. |
Required |
||
|
Object (as follows). |
Required |
||
|
List of strings, each a Secret name (or ID). |
Optional |
||
|
A specific Decodable-supported Flink version (string). |
Required |
||
|
List of strings, each a Flink metric to expose in the |
Optional |
||
|
Whether the pipeline is active or not. Boolean Default: |
Optional |
||
|
The task size for the pipeline. Valid values: Default: |
Optional |
||
|
The number of tasks to run for the pipeline. Default: |
Optional |
||
|
For each stream, the start position from which to read when starting the pipeline for the first time, or after a Specify the stream name and its starting position. Valid values: Default: Specify the stream name and its starting position. For example:
yaml
|
Optional |
||
|
When starting the pipeline for the first time, or after a For example:
yaml
|
Optional |
Secret
A template for a secret definition:
---
kind: secret
metadata:
name: <name_your_secret>
description: <description>
tags:
<tag_key_1>: <tag_value_1>
spec_version: v1
spec:
# Use _one_ of the following value_* fields.
value_env_var: MY_PASSWORD
value_file: ./secret.txt
value_literal: "don't-tell-123"
A Decodable Secret resource stores a plaintext value (such as a password) securely, so that it may be used by Connections or Custom Pipelines.
A Connection references a Secret by name from a (secret-type) property
value.
The apply
CLI command can set this plaintext value from an environment variable, a file, or a literal value, via special spec
fields.
These fields can only be used to write the plaintext value: the query command output for a Secret will not include them. The fields aren’t considered part of the Secret resource. Instead they act as directives to the CLI apply command to write the plaintext value to the secure Secret storage in your Decodable account.
|
Field | Description | ||
---|---|---|---|
|
The name of an environment variable containing the plaintext value. This is recommended for a simple plaintext value string such as a password. Note that not all binary values can be stored in an environment variable, and that some shell-based techniques can strip newlines when setting an environment variable. The |
||
|
A path (relative to the YAML file’s directory) to a local file containing the plaintext value. This is recommended for a multiline string (such as a private key) or binary plaintext value. The file contents are used verbatim. Note that this includes any trailing newlines. The |
||
|
A string with the literal plaintext value.
This field may be appropriate for private ad-hoc use, and for use with generated YAML. The other two fields are recommended for most CI/CD use. |
For API users: These fields are specially handled by the CLI apply command; the underlying /apply API doesn’t directly support them.
|
Secret definition without plaintext value
When no spec.value_*
field is present—as in query
command output—no plaintext value is written for the Secret resource on apply
.
In this case there is no spec
for the Secret definition, but spec_version: v1
is still required.
For a Secret created this way, the plaintext value may be manually set after apply
, as a separate step.
This can be done via the Decodable UI or CLI (decodable secret write <id> --value '<value>'
). It need only be done once per Secret.
Alternately, a spec.value_*
field may be added and apply
repeated.
If the Secret resource already exists and no spec.value_*
field is provided then the existing plaintext value will remain unchanged. This may be appropriate for certain use cases.