Create and update resources You can apply Decodable resource definitions to your Decodable account, using a YAML file and the decodable apply CLI command. See Declarative Resource Management overview if you are new to declarative. Applying a set of resources is idempotent and atomic: if anything goes wrong, no changes are made. If an apply succeeds, you can be sure the new state is valid, and your Decodable resources are in sync with your YAML file(s). A successful apply command response indicates which resources were created, updated, or remained unchanged. When you run the apply command to create or manage resources, the order of appearance of those resources in the YAML file does not matter. The apply command always prioritizes dependency order. This means that secret and stream resources are always created before the connection and pipeline resources that depend on them. The response shown after applying always presents the results in the same order as the input resources. The apply process stops on the first error it finds, saving no changes to any resource. In this case, the response provides details about the error, rather than listing results for each input resource. The apply command operates atomically: either all changes succeed simultaneously, or no changes are made. Audit events are always sent to the _events stream for any created or updated resources (except in --dry-run mode). Types of operation The decodable apply command supports creating and updating resources. Resources are identified by kind and metadata.name. To rename an existing resource add the id of the existing resource to the metadata field and change the name as required. See Renaming a resource. Resources cannot be deleted using the apply command. To delete resources created by an apply, use the query command with the delete operation. How to use the apply command You’ll need the Decodable CLI (version 1.14.0 or later). Step 1: Create resource definitions To start, you’ll need a YAML file to contain your resource definitions. In the examples below, we call it resources.yaml. Each resource definition in the file is its own YAML document. It must begin with a line of three hyphens ---, and contain the following fields: --- kind: <resource_type> metadata: name: <name_your_resource> description: <resource_description> spec_version: v1 spec: <resource_specifications> For a full explanation of each field see the definition reference. To get all resources already in your account using query: decodable query --export > resources.yaml A complete example of a YAML file with resource definitions for a Decodable pipeline to move data between two Apache Kafka topics can be found below. Step 2: Apply the resource definitions Call decodable apply with the name of the file(s) holding the resource definition(s). decodable apply resources.yaml The decodable apply command accepts multiple input files. This can be especially useful with shell globbing (with *). For example, decodable apply resources/*.yaml applies all resources in all YAML files in the resources directory. If you specify - in place of an input file name then apply will read from stdin, which is useful when used with other shell commands. The response contains a set of YAML documents for each resource in the apply input, identifying the resource (by kind, name, and id), and showing the apply result for that resource ( created, updated, or unchanged). Here is an example of a successful response after applying: --- kind: secret name: my-secret id: 3a9ed7ce result: unchanged --- kind: connection name: my-kafka-source-connection id: a7b322d4 result: created --- kind: stream name: kafka-stream id: b5f8c2de result: updated Certain Decodable CLI actions (such as activating a resource and setting values for secrets) require a resource ID so retaining this output is recommended. Optionally preview changes with the --dry-run option Using the --dry-run option with the apply command simulates all changes defined in a YAML file without making any actual changes, allowing you to preview potential creations or updates and identify any errors in your resource definitions. For example: decodable apply resources.yaml --dry-run The resulting output is the same, except that each output YAML doc will also have: dry_run: true Step 3: Set secret plaintext values If the YAML contains new secrets, set their plaintext secret values. Decodable CLI Decodable Web Get the id value from the apply output. Write the plaintext for this secret: decodable secret \ write <secret_id> \ --value <secret_value> From the Secrets page, find the secret. Select the ellipsis icon (…) > Set Value. Enter the secret value and select Set. You need only do this once for each new Secret resource. Step 4: Activate connections and pipelines Connections and pipelines cannot be activated declaratively. Activate them manually to start processing data. Decodable CLI Decodable Web Connections: decodable connection \ activate <connection_id> Pipelines: decodable pipeline \ activate <pipeline_id> Depending on the kind of resource that you want to activate, navigate to the Connections or Pipelines page. Select the resource that you defined earlier. From the resource’s Overview page, select Start. Renaming a resource Get the ID for the resource to be renamed: Using the Decodable CLI query command: decodable query --keep-ids resources.yaml or decodable query --keep-ids --name '<resource-name>' Using the Decodable CLI non-declaratively: decodable <resource-kind> list Using Decodable Web: Find the resource on the Connections, Streams, Pipelines, or Secrets page. Select the ellipsis (…) next to the resource, and select Copy ID. In your YAML add an id field to the metadata section: --- kind: <resource_type> metadata: id: <existing_resource_id> name: <updated_resource_name> description: <resource_description> spec_version: v1 spec: <resource_specifications> Apply the changes. decodable apply resources.yaml Once applied, make sure to remove the metadata.id field from your YAML file. The new name will now properly identify the resource. Example: A Decodable pipeline to move data between two Apache Kafka topics Let’s look at an example YAML file that creates the Decodable resources required to establish a connection between two Apache Kafka topics. It defines four resources: a secret containing our Apache Kafka password, a stream, and two connections: one to receive data from one Kafka topic and another to send data to another Apache Kafka topic. Create the required resources. In this step, we’ll create a secret for our connection, configure a source and sink connection, and create a stream to transport data between these connections. --- kind: secret metadata: name: kafka-sasl-password description: Password for Kafka SASL username spec_version: v1 --- kind: connection metadata: name: My-Kafka-Connection description: A connection to my Kafka topic spec_version: v1 spec: connector: kafka # The name of the connector. type: source # The type of connector. Enter source if your connector receives data or sink if your connector sends data. properties: # The properties of the connector that you want to use. Refer to the connector's documentation for property names and their valid values. bootstrap.servers: <broker_list> topic: <source_topic_name> value.format: json security.protocol: SASL_SSL sasl.mechanism: SCRAM-SHA-256 sasl.username: <username> sasl.password: kafka-sasl-password # The name of the secret defined earlier. stream_name: my-kafka-stream # The name of the stream that you want this connector to send data to. schema_v2: # The schema of the connection. This must match the schema of the stream that it is connected to exactly, including any constraints like watermarks or primary key fields. fields: - kind: physical name: field1 type: string - kind: physical name: field2 type: string - kind: physical name: field3 type: string --- kind: stream metadata: name: my-kafka-stream description: An example Kafka stream spec_version: v1 spec: schema_v2: fields: - kind: physical name: field1 type: string - kind: physical name: field2 type: string - kind: physical name: field3 type: string --- kind: connection metadata: name: Destination-Kafka-Connection description: A connection to my Kafka topic spec_version: v1 spec: connector: kafka type: sink properties: bootstrap.servers: <broker_list> topic: <destination_topic_name> value.format: json security.protocol: SASL_SSL sasl.mechanism: SCRAM-SHA-256 sasl.username: <username> sasl.password: kafka-sasl-password # The name of the secret defined above. stream_name: my-kafka-stream # The name of the stream that you want this connector to receive data from. schema_v2: fields: - kind: physical name: field1 type: string - kind: physical name: field2 type: string - kind: physical name: field3 type: string A response containing the resources that were created or modified is shown. Apply the YAML definitions to create the resources. decodable apply resources.yaml If the YAML contains new secrets, add the plaintext secret values. Using Decodable Web: From the Secrets page, find the secret that you defined earlier. Select the ellipsis icon (…) > Set Value. Enter the secret value and select Set. Using the Decodable CLI: decodable secret write <secret_id> --value <secret_value> Connections and pipelines are not activated by default. Activate them to start processing data. Using Decodable Web: Depending on the kind of resource that you want to activate, navigate to the Connections or Pipelines page. Select the resource that you defined earlier. From the resource’s Overview page, select Start. Using the Decodable CLI: Connections: decodable connection activate <connection_id> Pipelines: decodable pipeline activate <pipeline_id>