Using declarative definitions

Creating and managing Decodable resources declaratively using the decodable apply command involves the following steps. If you are new to declarative definitions, see the declarative overview.

  1. Create or update a YAML file containing resource definitions. Each resource definition in the file appears as a YAML document, which begins with a line of three hyphens ---.

    ---
    kind: <resource_type>
    metadata:
      name: <name_your_resource>
      description: <resource_description>
    spec_version: v1
    spec:
      <resource_specifications>
    Field Required or optional Description

    kind

    Required

    The kind of resource. One of: secret, connection, pipeline, stream.

    metadata

    Required

    Values that identify or describe the resource, including name and description text strings. The name value is mapped to an ID when the YAML is applied.

    spec_version

    Required: v1

    Determines spec format and processing logic for this kind. Currently always v1.

    spec

    Required, except with kind: secret

    The resource’s specifications. The format of the spec object is different for each Decodable resource kind. It contains nested fields specific to that resource.

    See: Declarative definition reference.

  2. Apply the YAML file(s) to create or update resources. Resources are identified by name metadata.name. When applying the YAML file, existing resources with matching names and types are updated, while new resources are created. For more information about how the apply command works, see Understanding the apply command.

    decodable apply resources.yaml

    The following is an example of a successful response after applying.

    ---
    kind: secret
    name: My-Secret
    id: 3a9ed7ce
    result: unchanged
    ---
    kind: connection
    name: My-Kafka-Source-Connection
    id: a7b322d4
    result: created
    ---
    kind: stream
    name: Kafka-Stream
    id: b5f8c2de
    result: updated
    ---
    kind: pipeline
    name: Parsing-Pipeline
    id: 9a4b5e9d
    result: created

    From the response, you can view the id of each resource. These IDs are necessary for certain Decodable CLI actions like activating a resource. Additionally, the response displays the status of each resource post-application: created, unchanged, or updated.

  3. If the YAML contains new secrets, add the plaintext secret values.

    • Using Decodable Web:

      1. From the Secrets page, find the secret that you defined earlier.

      2. Select the ellipsis icon (…​) > Set Value.

      3. Enter the secret value and select Set.

    • Using the Decodable CLI:

      decodable secret write <secret_id> --value <secret_value>
  4. Connections and pipelines are not activated by default. Activate them to start processing data.

    • Using Decodable Web:

      1. Depending on the kind of resource that you want to activate, navigate to the Connections or Pipelines page.

      2. Select the resource that you defined earlier.

      3. From the resource’s Overview page, select Start.

    • Using the Decodable CLI:

      • Connections:

        decodable connection activate <connection_id>
      • Pipelines:

        decodable pipeline activate <pipeline_id>
Resources cannot be deleted using the apply command. To delete resources, use Decodable Web or run decodable <resource> delete <resource_id> in the Decodable CLI.

Understanding the apply command

When you run the apply command to create or manage resources, the order of appearance of those resources in the YAML file does not matter. The apply command always prioritizes dependency order. This means that secret and stream resources are always created before the connection and pipeline resources that depend on them. The response shown after applying always presents the resources in the same order as the input resources.

The apply process stops on the first error it finds, saving no changes to any resource. In this case, the response provides details about the error, rather than listing results for each input resource. The apply command operates atomically: either all changes succeed simultaneously, or no changes are implemented.

The decodable apply command accepts multiple input files. This can be especially useful with shell globbing (with *). For example, decodable apply resources/* applies all resources in all files in the resources directory.

Audit events are always sent to the _events stream for any created or updated resources, unless in --dry-run mode.

Preview changes with the --dry-run option

You can see the results of running the apply command by using the --dry-run option. The --dry-run option causes apply to attempt all changes defined in a given YAML file without actually saving those changes. This allows you to see what would be created or updated, or discover any errors in your YAML specifications, without affecting your real Decodable resources.

For example:

decodable apply resources.yaml --dry-run

A successful response looks like the following:

---
kind: secret
name: My-Secret
id: 3g7k9p5a
result: unchanged
dry_run: true
---
kind: connection
name: My-Kafka-Source-Connection
id: a7b322d4
result: updated
dry_run: true
---
kind: stream
name: Kafka-Stream
id: b5f8s2lp
result: updated
dry_run: true

Renaming a resource

If you want to change the metadata.name for an existing resource using apply, do the following.

  1. Add an id field to metadata that contains the ID of the existing resource. For example:

    ---
    kind: <resource_type>
    metadata:
      id: <id>
      name: <updated_resource_name>
      description: <resource_description>
    spec_version: v1
    spec:
      <resource_specifications>
  2. To get the ID for a specific resource:

    1. Using Decodable Web:

      1. Find the resource on the Connections, Streams, Pipelines, or Secrets page.

      2. Select the ellipsis (…​) next to the resource, and select Copy ID.

    2. Using the Decodable CLI:

      decodable <resource> list
  3. Apply the changes.

    decodable apply resources.yaml
  4. Remove the metadata.id field from your YAML file. The new name will now properly identify the resource.