Create and update resources

You can apply Decodable resource definitions to your Decodable account, using a YAML file and the decodable apply CLI command.

If you are new to declarative resource management in Decodable see the overview.

Applying a set of resources is idempotent and atomic: if anything goes wrong, no changes are made. If an apply succeeds, you can be sure the new state is valid, and your Decodable resources are in sync with your YAML files. A successful apply command response indicates which resources were created, updated, or remained unchanged.

When you run the apply command to create or manage resources, the order of appearance of those resources in the YAML file doesn’t matter. The apply command always prioritizes dependency order. This means that secret and stream resources are always created before the connection and pipeline resources that depend on them. The response shown after applying always presents the results in the same order as the input resources.

The apply process stops on the first error it finds, saving no changes to any resource. In this case, the response provides details about the error, rather than listing results for each input resource. The apply command operates atomically: either all changes succeed simultaneously, or no changes are made.

Audit events are always sent to the _events stream for any created or updated resources (except in --dry-run mode).

Types of operation

The decodable apply command supports creating and updating resources. Resources are identified by kind and metadata.name.

To rename an existing resource add the id of the existing resource to the metadata field and change the name as required. See Renaming a resource.

Resources can’t be deleted using the apply command. To delete resources created by an apply, use the query command with the delete operation.

How to use the apply command

You’ll need the Decodable CLI (version 1.14.0 or later).

Step 1: Create resource definitions

To start, you’ll need a YAML file to contain your resource definitions. In the examples below, we call it resources.yaml.

Each resource definition in the file is its own YAML document. It must begin with a line of three hyphens ---, and contain the following fields:

---
kind: <resource_type>
metadata:
  name: <name_your_resource>
  description: <resource_description>
spec_version: v1
spec:
  <resource_specifications>

For a full explanation of each field see the definition reference.

To get all resources already in your account using query:

decodable query --export > resources.yaml

A complete example of a YAML file with resource definitions for a Decodable pipeline to move data between two Apache Kafka topics can be found below.

Step 2: Apply the resource definitions

Call decodable apply with the name of one or more files holding the resource definitions.

decodable apply resources.yaml

The decodable apply command accepts multiple input files. This can be especially useful with shell globbing (with *). For example, decodable apply resources/*.yaml applies all resources in all YAML files in the resources directory.

If you specify - in place of an input file name then apply will read from stdin, which is useful when used with other shell commands.

The response contains a set of YAML documents for each resource in the apply input, identifying the resource (by kind, name, and id), and showing the apply result for that resource ( created, updated, or unchanged). Here is an example of a successful response after applying:

---
kind: secret
name: my-secret
id: 3a9ed7ce
result: unchanged
---
kind: connection
name: my-kafka-source-connection
id: a7b322d4
result: created
---
kind: stream
name: kafka-stream
id: b5f8c2de
result: updated
Certain Decodable CLI actions (such as activating a resource and setting values for secrets) require a resource ID so retaining this output is recommended.

Optionally preview changes with the --dry-run option

Using the --dry-run option with the apply command simulates all changes defined in a YAML file without making any actual changes, allowing you to preview potential creations or updates and identify any errors in your resource definitions.

For example:

decodable apply resources.yaml --dry-run

The resulting output is the same, except that each output YAML doc will also have:

dry_run: true

Step 3: Set secret plaintext values

If the YAML contains new secrets, set their plaintext secret values.

Decodable CLI Decodable Web
  1. Get the id value from the apply output.

  2. Write the plaintext for this secret:

    decodable secret \
     write <secret_id> \
     --value <secret_value>
  1. From the Secrets page, find the secret.

  2. Select the ellipsis icon (…​) > Set Value.

  3. Enter the secret value and select Set.

You need only do this once for each new Secret resource.

Step 4: Activate connections and pipelines

Connections and pipelines can’t be activated declaratively. Activate them manually to start processing data.

Decodable CLI Decodable Web
  • Connections:

    decodable connection \
      activate <connection_id>
  • Pipelines:

    decodable pipeline \
      activate <pipeline_id>
  1. Depending on the kind of resource that you want to activate, navigate to the Connections or Pipelines page.

  2. Select the resource that you defined earlier.

  3. From the resource’s Overview page, select Start.

Renaming a resource

  1. Get the ID for the resource to be renamed:

    • Using the Decodable CLI query command:

      decodable query --keep-ids resources.yaml

      or

      decodable query --keep-ids --name '<resource-name>'
    • Using the Decodable CLI non-declaratively:

      decodable <resource-kind> list
    • Using Decodable Web:

      1. Find the resource on the Connections, Streams, Pipelines, or Secrets page.

      2. Select the ellipsis (…​) next to the resource, and select Copy ID.

  2. In your YAML add an id field to the metadata section:

    ---
    kind: <resource_type>
    metadata:
      id: <existing_resource_id>
      name: <updated_resource_name>
      description: <resource_description>
    spec_version: v1
    spec:
      <resource_specifications>
  3. Apply the changes.

    decodable apply resources.yaml
    Once applied, make sure to remove the metadata.id field from your YAML file. The new name will now identify the resource.

Example: A Decodable pipeline to move data between two Apache Kafka topics

Let’s look at an example YAML file that creates the Decodable resources required to establish a connection between two Apache Kafka topics. It defines four resources: a secret containing our Apache Kafka password, a stream, and two connections: one to receive data from one Kafka topic and another to send data to another Apache Kafka topic.

  1. Create the required resources. In this step, we’ll create a secret for our connection, configure a source and sink connection, and create a stream to transport data between these connections.

    ---
    kind: secret
    metadata:
      name: kafka-sasl-password
      description: Password for Kafka SASL username
    spec_version: v1
    ---
    kind: connection
    metadata:
      name: My-Kafka-Connection
      description: A connection to my Kafka topic
    spec_version: v1
    spec:
      connector: kafka # The name of the connector.
      type: source # The type of connector. Enter source if your connector receives data or sink if your connector sends data.
      properties: # The properties of the connector that you want to use. Refer to the connector's documentation for property names and their valid values.
        bootstrap.servers: <broker_list>
        topic: <source_topic_name>
        value.format: json
        security.protocol: SASL_SSL
        sasl.mechanism: SCRAM-SHA-256
        sasl.username: <username>
        sasl.password: kafka-sasl-password # The name of the secret defined earlier.
      stream_name: my-kafka-stream # The name of the stream that you want this connector to send data to.
      schema_v2: # The schema of the connection. This must match the schema of the stream that it's connected to exactly, including any constraints like watermarks or primary key fields.
        fields:
          - kind: physical
            name: field1
            type: string
          - kind: physical
            name: field2
            type: string
          - kind: physical
            name: field3
            type: string
    ---
    kind: stream
    metadata:
      name: my-kafka-stream
      description: An example Kafka stream
    spec_version: v1
    spec:
      schema_v2:
        fields:
          - kind: physical
            name: field1
            type: string
          - kind: physical
            name: field2
            type: string
          - kind: physical
            name: field3
            type: string
    ---
    kind: connection
    metadata:
      name: Destination-Kafka-Connection
      description: A connection to my Kafka topic
    spec_version: v1
    spec:
      connector: kafka
      type: sink
      properties:
        bootstrap.servers: <broker_list>
        topic: <destination_topic_name>
        value.format: json
        security.protocol: SASL_SSL
        sasl.mechanism: SCRAM-SHA-256
        sasl.username: <username>
        sasl.password: kafka-sasl-password # The name of the secret defined above.
      stream_name: my-kafka-stream # The name of the stream that you want this connector to receive data from.
      schema_v2:
        fields:
          - kind: physical
            name: field1
            type: string
          - kind: physical
            name: field2
            type: string
          - kind: physical
            name: field3
            type: string

    A response containing the resources that were created or modified is shown.

  2. Apply the YAML definitions to create the resources.

    decodable apply resources.yaml
  3. If the YAML contains new secrets, add the plaintext secret values.

    • Using Decodable Web:

      1. From the Secrets page, find the secret that you defined earlier.

      2. Select the ellipsis icon (…​) > Set Value.

      3. Enter the secret value and select Set.

    • Using the Decodable CLI:

      decodable secret write <secret_id> --value <secret_value>
  4. Connections and pipelines aren’t activated by default. Activate them to start processing data.

    • Using Decodable Web:

      1. Depending on the kind of resource that you want to activate, navigate to the Connections or Pipelines page.

      2. Select the resource that you defined earlier.

      3. From the resource’s Overview page, select Start.

    • Using the Decodable CLI:

      • Connections:

        decodable connection activate <connection_id>
      • Pipelines:

        decodable pipeline activate <pipeline_id>