Query existing resources If you are new to declarative resource management in Decodable see the overview. The decodable query CLI command fetches, or bulk-modifies, resources in your Decodable account. With no arguments, query fetches all user resources in your Decodable account. You would typically use filters to select a subset of resources from your Decodable account. The output of query can be used as input to decodable apply, as well as exported for committing to source control as the canonical copy of your resource definitions. You can apply operations to the resources selected by query, such as deleting or restarting them. The query command outputs a series of resource definitions in YAML format. It adds a status section which contains non-writable fields managed by Decodable for the resource. Here is an example of the output from a query command for a pipeline: decodable query --name example_pipe --kind pipeline --- kind: pipeline metadata: name: example_pipe description: Perform example processing spec_version: v2 spec: type: SQL sql: | INSERT INTO example_out SELECT lower(s) FROM example_in execution: active: true task_count: 1 task_size: M status: version: 1 is_latest: true active_version: 1 latest_version: 1 create_time: 2024-07-23T20:57:58.760+00:00 last_activated_time: 2024-07-23T21:45:47.276+00:00 execution: state: RUNNING The status section is ignored by the apply command, so the output from query can serve directly as input to decodable apply. Here’s an example of copying all resources that begin with epos- from one Decodable account to another by piping the output from query as input to apply (using - to denote reading from stdin): decodable --profile dev query --name "epos-*" | \ decodable --profile staging apply - Read more about using multiple profiles with the CLI here. Command reference You’ll need the Decodable CLI (version 1.20.0 or later). decodable query The query command (and corresponding API endpoint) supports many options, all optional. Most may be specified via command line (CLI) options, some of which allow a short form. All may be given via query specifier YAML. Each query is controlled with a query specifier object. The query command builds the specifier for you based on the given command line arguments. You can also provide the query specifier directly as YAML. CLI options that take <strings> allow multiple values, either by repeating the option or using a comma-separated (,) list. short long option specifier field description -y --specifier-yaml <yaml> Provide (base) query specifier as inline YAML. -f --specifier-file <file name> Provide (base) query specifier as a YAML file. If the file name is -, standard input (stdin) is used. Note that this is different from providing a file name on its own, not using this option. -X --operation <string> operation string Query operation: one of: get (default): Include associated values in each resource: schema, properties, etc. list : Include only scalar values. This can be faster in some cases. activate : Activate all matched resources. deactivate : Deactivate all matched resources. delete : Delete all matched resources. restart : Restart all matched resources that are currently active. reset-state : Reset the state of matched resources. -k --kind <strings> kinds array of string Include in response only resources whose kind is one of these values. -n --name <strings> names array of string Include in response only resources whose name matches one of these values. Values for this option may include the * wildcard, which matches any string in a name. For example: demo_*. -i --id <strings> ids array of string Include in response only resources whose id is one of these values. <file name> (or - to read from stdin) Accepts multiple. resources array of objects, each with kind and name string fields Include in response only resources whose kind and name equal those in one of these values. Sort response resources to match. Files specified must be in apply format. The kind and metadata.name is taken from each YAML doc in each file. --active <string> value: true or false active boolean When present, include in response only Connection and Pipeline resources that are: When true: active. When false: not active. --stable <string> value: true or false stable boolean When present, include in response only Connection and Pipeline resources that are: When true: stable. When false: not stable. --tag-expression <string> Value: a SQL expression tag_expression string Include in response only resources whose tags match the given SQL expression. --export or --no-status no_status boolean (Default: false) Return resources without status section, suitable for export for source control. --no-spec no_spec boolean (Default: false) Return resources without spec section. -M --metadata-only Combines --no-spec, --no-status, --operation=list. Only kind and the metadata section are retained in the response resources. Useful for a concise look at which resources are selected by a set of query options (specifier). --keep-ids keep_ids boolean (Default: false) Return resources with IDs in metadata and spec sections. --include-immutables include_immutables boolean (Default: false) Include in the response built-in immutable resources, such as _events and _metrics Streams. Important: These resources can’t be apply'd (or otherwise changed). Not appropriate for export to source control or apply. spec_versions object: • key: kind value • value: spec_version value (per-kind) Select the spec_version for one or more kinds. Allows your custom tooling to choose the contract (object shape) for spec sections of resource definition YAML docs in the query response. Note that spec shapes won’t be compatible across spec_versions, and spec_version is per-kind. The default spec_version is the latest for each kind. -S --output-specifier Don’t query. Output (computed) query specifier YAML instead. --stabilize Wait (poll) until all matched resources are stable. May be used without an operation, or with activate, deactivate, or restart. See: Waiting for resources to stabilize. --timeout <string> Value: a duration Fail --stabilize process after given timeout duration. Duration string is specified as a sequence of numbers, each with a unit suffix (h, m, s), for example: 90s, 1h30m. Resource definition files as filter In addition to command line options, query accepts optional input file names. These must be YAML files with resource definitions, in apply input / query output format. If a file name is -, standard input (stdin) is used. When file names are provided, results are: filtered to include only resources in the specified files (by kind and name); sorted to match the order of resources in the specified files; This can be useful after an apply to check the status of applied resources: decodable apply resources.yaml decodable query resources.yaml File names and other filter options may be used together. If other filter options are given, all filters must match: only resources defined in the given files and matching all other given filters will be included in the results. Shell globbing (with *) can be an effective way to provide a set of input files. For example, decodable query resources/*.yaml will query the state of all resources in all YAML files in the resources directory. Note that the -f/--specifier-file option also takes a file name, but this is in a different format and serves a different purpose. See the options table. Resource operations Resources that are selected by query can have operations performed on them using the --operation (or -X) argument. Since query has a powerful filter capability this is a great way to script the management of Decodable resources. There are two non-mutating operations: get and list. They return, for each matched resource, a YAML document with the full resource definition, including the status section. All other operations are mutating, and are atomic: they either mutate all matched resources successfully or fail with an error, changing nothing. Mutating operations return, for each matched resource, a YAML document with the resource definition for that resource. The document doesn’t include the status section. Get By default, decodable query performs --operation get which returns the full resource definition and status for each resource matching the optional filter. For example: # These two commands have the same effect decodable query --kind connection decodable query --kind connection --operation get List Similar to get, but include only scalar values. This can be faster in the case of a large number of resources with many complex fields such as schema and stream_mappings. For example: decodable query --kind connection --operation list Restart Using --operation restart will restart any matched resource that’s currently active. It won’t activate a resource that’s inactive, even if the resource matches the query filter. For example: decodable query --kind connection --operation restart See --stabilize for details of making the CLI wait until restart has completed (or, optionally, timed out). Reset state The reset-state operation is permanent. It causes a connection or pipeline to "start over" from a different place in the input, which can cause records to be skipped or reprocessed. Using --operation reset-state will mark any matched resource to start from an initial state. If the resource is currently active, it will be restarted. This can be useful for reprocessing records, or skipping to the latest point of the source stream and ignoring records that haven’t been processed yet. The initial state from which the resource will start is configured in its resource definition. For sink connections, specify initial_start_position in the resource definition. For pipelines, specify initial_start_positions (per stream) or initial_snapshot_id in the resource definition. If none of the above options are used the resource will start from latest position on next start after reset-state. For example: decodable query --kind connection --operation reset-state The reset-state operation requires at least one filter, for safety. If you really want to reset state for all Pipelines and Connections in your Decodable account, use: --name '*'. Delete The delete operation is a powerful tool. Deletion is permanent. Use --operation delete operation to remove matched resources. For example, to delete all resources whose names start with demo_: decodable query --name 'demo_*' --operation delete The delete operation requires at least one filter, for safety. If you really want to delete all resources in your Decodable account, use: --name '*'. To use delete in a safe manner, try your query filters first without the delete operation, inspect the output, then add the --operation delete option. Even safer: Redirect the fetch output to a file, inspect the file, then use that file as input for the delete. This ensures that only the resources in that file will be deleted, not any additional resources that may have been added after the query fetch that match the same filters. For example: decodable query --name 'demo_*' > to_delete.yaml # ... review, inspect ... decodable query --operation delete to_delete.yaml Activate and deactivate Use --operation activate to activate (start) matching resources. Use --operation deactivate to deactivate (stop) them. For example, to activate and then deactivate all resources whose names start with demo_: decodable query --name 'demo_*' --operation activate decodable query --name 'demo_*' --operation deactivate These are idempotent operations that affect only resources that can be active (connections and pipelines), with no effect on other resources (secrets and streams). An activate on an active resource or deactivate on an inactive resource has no effect, and isn’t an error. To actually restart active resources, use operation restart. See --stabilize for details of making the CLI wait until activate or deactivate has completed (or, optionally, timed out). Filtering resources The query command can filter resources (server-side) in four distinct ways, which may be combined. For example, to fetch all resources whose names start with demo_: decodable query --name 'demo_*' You can filter resources by: kind name (with * wildcard) id Per-resource kind and name, via input resource files resource execution state (active, stable) If you want to filter the results by conditions other than those available, you can do this with YAML post-processing. See also: Complete query filter examples. Options reference. YAML post-processing. Filter examples Names matching one or more * wildcard patterns, such as a prefix: decodable query --name 'demo_*' --name '*_test_*' By combining the two strings into a comma-separated list this can also be written as: decodable query --name 'demo_*,*_test_*' Specific kinds of resources: decodable query --kind 'connection,pipeline' A specific set of IDs (across kinds): decodable query --id '83ed9491,ca64352b,044b98ff' All resources defined in one or more apply-format YAML files. This is useful for checking the status of apply'd resources. decodable query resources1.yaml resources2.yaml All resources with tags matching a SQL expression. decodable query --tag-expression "environment='production'" Combining filters Each filter is optional, but all given filters must match. Here is an example that will fetch all connection and pipeline resources whose names start with demo_, but only with the given IDs, and only when appearing in resource.yaml. Example query usage that combines filters decodable query \ --kind 'connection,pipeline' \ --name 'demo_*' \ --id '83ed9491,ca64352b,044b98ff' \ resource.yaml Filtering by resource execution state You can filter resources based on their active or stable status, via the active and stable query filters. If provided, the filter filters must be given with an explicit true or false value. For example, to see all resources in the account that aren’t stable: decodable query --stable=false To see all resources in file resources.yaml that are trying (or failing) to start: decodable query resources.yaml --active=true --stable=false Filter Value Effect active true Include only active resources. false Include only inactive resources. stable true Include only stable resources. false Include only unstable resources. Only connection and pipeline resources can be activated, thus only they can be matched by these filters. Filtering by resource tags You can filter resources by their associated tags using the --tag-expression/-t option with a SQL expression. A tag is a key-value pair defined on resources such as streams, connections, pipelines, and secrets. In the tag-matching SQL expression syntax, an identifier corresponds to a tag key and a single-quoted string corresponds to a tag value (as in key1='value1'). A key identifier may be quoted as needed with backticks (as in `my-key`). Multiple tag conditions can be combined using logical operators like AND and OR. To filter for the presence of a key, regardless of value, specify only the key in the expression. Other filters (such as --name or --kind) can be combined with the tag filter. Tag filtering supports: Key presence matches Key-value match operators: =, IN, LIKE Logical operators: AND, OR, NOT Parentheses for explicit precedence Remember, in a tag expression a tag value must be enclosed in single quotes. On the command line, enclose the entire expression in double quotes and escape any backticks. Here are some examples of filtering by tag: Resources that have a tag with the key env and value prod: decodable query --tag-expression "env='prod'" Resources that are in production for a set of projects: decodable query --tag-expression "env='prod' AND project IN ('proto', 'revenue')" Resources approved in 2024: decodable query --tag-expression "approved_at LIKE '2024-%'" Connections that have the env tag and a name matching a given pattern: decodable query --tag-expression "env" --name 'ingest*' --kind connection Waiting for resources to stabilize While mutating operations (such as activate and restart) return immediately, it can take a few seconds to a few minutes (or more) for the changes to become fully effective—that is, for the resources to become stable. A stable resource is doing what you want: running if active, stopped if not. A stable resource is neither still transitioning toward its target state nor stuck in a broken state. The query --stabilize option will wait until all matched resources become stable, printing resource state changes (to stderr) along the way. This option may be used either with no operation or with any of these operations: restart, reset-state, activate, deactivate. For example, to await stability of all resources named in file resources.yaml: decodable query resources.yaml --stabilize A resource that’s failing to start (or stop) with a non-transient problem—perhaps due to some misconfiguration—will never become stable (without intervention). Thus, a query --stabilize process matching such a resource will never terminate, unless a timeout is given. To timeout after a duration, add the --timeout=duration option, where the duration is specified as a sequence of numbers, each with a unit suffix (h, m, s), for example: 90s, 1h30m. If all matched resources in the query don’t become stable within the timeout duration, the query command will fail with a non-zero exit code. Stop and start durations for pipelines with large working state can be long, as this state must be loaded or stored to durable storage. Effective timeout durations must be tuned against the actual subject resources, operating against real data. Using a timeout can be especially important for scripts or automated systems (as used in CI/CD), which should themselves fail when resources won’t correctly start or stop. For example, to activate all resources named in resources.yaml, then wait for them all to become stable for at most 2 minutes 30 seconds: decodable query resources.yaml --operation activate --stabilize --timeout=2m30s Example usage Fetch all user resources in your Decodable account: decodable query Exporting resources to source control Use the query command to export your Decodable resources for storing in a source control system like git. Maintaining resources via files in source control serves as the basis for "infrastructure-as-code" development practices. The apply command may be used to create or update resources to match the resource files in source control. This can be automated via CI/CD. Always use the --export (or --no-status) option when exporting for source control. This option suppresses the status section in query result resources. Although apply ignores the status section, status isn’t appropriate for resource definitions in source control. Never use --keep-ids when exporting for source control: IDs can’t be replicated to new resources (after a resource deletion or in a different account, for example), and so break repeatability. For example: decodable query --export > decodable_resources.yaml git add decodable_resources.yaml git commit -m "IaC FTW!" # obviously you use better commit messages than this ;) # ... elsewhere later ... git pull decodable apply decodable_resources.yaml Result modifier examples Export resources without the status section. This is useful for when you want to commit the resource definitions to source control (like git): decodable query --export You can use this to write it directly to a file, by redirecting stdout: decodable query --export > resources.yaml Note that --export is a synonym for --no-status. Get just the (runtime) status for resources (that’s, don’t return the full resource configuration spec): decodable query --no-spec Get just the resource kind and metadata (name, description) (excluding spec or status): decodable query --metadata-only Deleting resources from apply The delete operation can be used to remove a set of resources created or modified by the apply command: decodable apply resources.yaml # ... use these resources ... decodable query resources.yaml --operation delete Result order will match that of the resources in the input files. Note that any resources specified in files must (still) exist to be deleted. Specified resources that don’t exist are simply ignored. YAML post-processing Since query returns valid YAML to stdout, you can pipe this into any YAML command-line tool for further processing. The following examples use the yq tool. Show the query YAML output with colored syntax highlighting: decodable query | yq Count resources by kind: decodable query -M \ | yq eval-all ' [.] | group_by(.kind) | map({"kind": .[0].kind, "count": length})' Count connection resources by connector: decodable query -k connection -X list \ | yq eval-all ' [.] | group_by(.spec.connector) | map({"connector": .[0].spec.connector, "count": length})' Change a word prefix in any string value to a new prefix: from=staging_ to=production_ decodable query --export \ | yq ' (.. | select(tag == "!!str")) |= sub("\b'${from}'", "'${to}'")' Filter the list of connections for just those using a particular connector, such as iceberg: decodable query -k connection | \ yq eval-all 'select(.spec.connector=="iceberg")' You can extend the above example to parameterize it: connector=iceberg decodable query -k connection | \ yq eval-all 'select(.spec.connector=="'${connector}'")' Query specifier The query command will build up a query specifier based on the optional arguments provided. You can also provide the base specifier directly as YAML, which can be useful for scripts and tools. Any additional query CLI options are added to the given YAML specifier. The specifier is an object whose fields are shown in the options table above in the "specifier field" column. You can provide the specifier as inline YAML: decodable query -y '{kinds: [connection,pipeline]}' or as a YAML file: decodable query -f specifier.yaml Example query specifier YAML kinds: - connection - pipeline names: - demo_* - "*_tmp" ids: [83ed9491,ca64352b,044b98ff] resources: - kind: connection name: demo_1_input - kind: pipeline name: demo_1_processor no_status: true tag_expression: "environment='production'" To generate a query specifier file, run query with the options required, and the --output-specifier (or -S) option. Instead of running the query, it will return the computed query specifier. For example: decodable query --output-specifier --kind pipeline --name name1,name2 outputs: kinds: - pipeline names: - name1 - name2