Query existing resources The decodable query CLI command fetches or bulk-modifies resources in your Decodable account. It supports: Configurable resource filters and result modifiers. Resource export for source control. Resource activation and deactivation. Resource deletion. If you are new to declarative resource management in Decodable see the overview. You’ll need the Decodable CLI (version 1.17.0 or later). With no arguments, query fetches all user resources in your Decodable account: decodable query See query examples and options reference. Query output The query command outputs a series of resource definitions, each as a YAML doc that extends the apply input format with a status section. The status contains non-writable fields managed by Decodable for the resource. Example query output --- kind: pipeline metadata: name: example_pipe description: Perform example processing spec_version: v1 spec: sql: | INSERT INTO example_out SELECT lower(s) FROM example_in status: version: 1 is_latest: true type: SQL latest_version: 1 target_state: RUNNING actual_state: RUNNING requested_tasks: 2 actual_tasks: 2 create_time: 2024-04-26T17:19:35.311+00:00 The status section is ignored by the apply command, so the output from query can serve directly as input to decodable apply. Filtering resources The query command can filter resources (server-side) in four distinct ways, which may be combined. For example, to fetch all resources whose names start with demo_: decodable query --name 'demo_*' You can filter resources by: kind name (with * wildcard) id Per-resource kind and name, via input resource files active, stable: by resource state See: Complete query filter examples. Options reference. Exporting existing resources to source control Use the query command to export your Decodable resources for storing in a source control system like git. Maintaining resources via files in source control serves as the basis for "infrastructure-as-code" development practices. The apply command may be used to create or update resources to match the resource files in source control. This can be automated via CI/CD. Always use the --export (or --no-status) option when exporting for source control. This option suppresses the status section in query result resources. Although apply ignores the status section, status isn’t appropriate for resource definitions in source control. Never use --keep-ids when exporting for source control: IDs can’t be replicated to new resources (after a resource deletion or in a different account, for example), and so break repeatability. For example: decodable query --export > decodable_resources.yaml git add decodable_resources.yaml git commit -m "IaC FTW!" # obviously you use better commit messages than this ;) # ... elsewhere later ... git pull decodable apply decodable_resources.yaml Activating and deactivating resources Use the query command with the activate operation to activate (start) matching resources. Use the deactivate operation to deactivate (stop) them. These operations are compatible with any combination of query filters. The operation may be specified with --operation or -X. For example, to activate and then deactivate all resources whose names start with demo_: decodable query --name 'demo_*' --operation activate decodable query --name 'demo_*' --operation deactivate These are idempotent operations that affect only resources that can be active (connections and pipelines), with no effect on other resources (secrets and streams). An activate on an active resource or deactivate on an inactive resource has no effect, and isn’t an error. Waiting for resources to stabilize While the activate and deactivate operations return immediately, it can take a few seconds to a few minutes (or more) for the changes to become effective—that is, for the resources to become stable. A stable resource is doing what you want: running if active, stopped if not. A stable resource is neither still transitioning toward its target state, nor stuck in a broken state. The query --stabilize option will wait until all matched resources become stable, printing resource state changes (to stderr) along the way. This option may be used either with no operation, or with either operation activate or deactivate. For example, to await stability of all resources named in file resources.yaml: decodable query resources.yaml --stabilize A resource that’s failing to start (or stop) with a non-transient problem—perhaps due to some misconfiguration—will never become stable (without intervention). Thus, a query --stabilize process matching such a resource will never terminate, unless a timeout is given. To timeout after a duration, add the --timeout=duration option, where the duration is specified as a sequence of numbers, each with a unit suffix (h, m, s), for example: 90s, 1h30m. If all matched resources in the query don’t become stable within the timeout duration, the query command will fail with a non-zero exit code. Stop and start durations for pipelines with large working state can be long, as this state must be loaded or stored to durable storage. Effective timeout durations must be tuned against the actual subject resources, operating against real data. Using a timeout can be especially important for scripts or automated systems (as used in CI/CD), which should themselves fail when resources won’t correctly start or stop. For example, to activate all resources named in resources.yaml, then wait for them all to become stable for at most 2 minutes 30 seconds: decodable query resources.yaml -X activate --stabilize --timeout=2m30s Querying by execution status You can filter resources based on their active or stable status, via the active and stable query filters. Each of these filters must be given with an explicit true or false value. Filter Value Effect active true Include only active resources. false Include only inactive resources. stable true Include only stable resources. false Include only unstable resources. Only connection and pipeline resources can be activated, thus only they can be matched by these filters. For example, to see all resources in the account that aren’t stable: decodable query --stable=false To see all resources in file resources.yaml that are trying (or failing) to start: decodable query resources.yaml --active=true --stable=false Restarting active resources Sometimes you need to restart active resources. The query command’s activate operation, however, has no effect on active resources: it doesn’t restart them. Instead, the query command can be used to restart a specific set of active resources using a series of steps: query the specific active resources into a new file. query deactivate using that file (with --stabilize). query activate using that file. This ensures that only the resources that were originally active are activated. If you simply deactivate and then activate a set of resources (matching the same query filters) without first capturing the active resources to a file, you would activate even matching resources that had not originally been active, which may not be what you want. For example, to restart only the already-active resources named with prefix demo_, waiting for them to fully stop and then fully start: decodable query -n 'demo_*' --active=true > /tmp/restart.yaml decodable query /tmp/restart.yaml -X deactivate --stabilize decodable query /tmp/restart.yaml -X activate --stabilize Consider using --timeout when doing this from a script or CI/CD. Deleting resources Use the query command with the delete operation to remove matched resources. This is compatible with any combination of query filters. For example, to delete all resources whose names start with demo_: decodable query --name 'demo_*' --operation delete Safe deletion The delete operation is a powerful tool. Deletion is permanent. To use delete in a safe manner, try your query filters first without the delete operation, inspect the output, then add the --operation delete (or -X delete) option. Even safer: Redirect the fetch output to a file, inspect the file, then use that file as input for the delete. This ensures that only the resources in that file will be deleted, not any additional resources that may have been added after the query fetch that match the same filters. For example: decodable query --name 'demo_*' > to_delete.yaml # ... review, inspect ... decodable query -X delete to_delete.yaml Understanding delete Like the apply command, the query delete operation is atomic: it either deletes all matched resources successfully or fails with an error, deleting nothing. The delete operation returns apply-style resource status format YAML, rather than resource definition format YAML returned by non-mutating (fetching) query operations. Currently, delete will fail if any matched runnable resources are RUNNING. This applies to connections and pipelines. To delete a resource set that includes such resources, deactivate them first. The delete operation requires at least one filter, for safety. If you really want to delete all resources in your Decodable account, use: --name '*'. Deleting resources from apply The query delete operation also removes a set of resources created or modified by the apply command: decodable apply resources.yaml # ... use these resources ... decodable query resources.yaml -X delete Result order will match that of the resources in the input files. Note that any resources specified in files must (still) exist to be deleted. Specified resources that don’t exist are simply ignored. Arguments guide and options reference The query command (and corresponding API endpoint) supports many options, all optional. Most may be specified via command line (CLI) options, some of which allow a short form. All may be given via query specifier YAML. Each query is controlled with a query specifier object. The query command builds the specifier for you based on the given command line arguments. You can also provide the query specifier directly as YAML. Resource definition files as filter In addition to command line options, query accepts optional input file names. These must be YAML files with resource definitions, in apply input / query output format. If a file name is -, standard input (stdin) is used. When file names are provided, results are: filtered to include only resources in the specified files (by kind and name); sorted to match the order of resources in the specified files; This can be useful after an apply to check the status of applied resources: decodable apply resources.yaml decodable query resources.yaml file names and other filter options may be used together. If other filter options are given, all filters much match: only resources defined in the given files and matching all other given filters will be included in the results. Shell globbing (with *) can be an effective way to provide a set of input files. For example, decodable query resources/yaml.* will query the state of all resources in all YAML files in the resources directory. Note that the -f/--specifier-file option also takes a file name, but this is in a different format and serves a different purpose. See table below. Options reference CLI options that take <strings> allow multiple values, either by repeating the option or using a comma-separated (,) list. short long option specifier field description -y --specifier-yaml <yaml> Provide (base) query specifier as inline YAML. -f --specifier-file <file name> Provide (base) query specifier as a YAML file. If the file name is -, standard input (stdin) is used. Note that this is different from providing a file name on its own, not using this option. -X --operation <string> operation string Query operation: one of: • get (default): Include associated values in each resource: schema, properties, etc. • list : Include only scalar values. This can be faster in some cases. • activate : Activate all matched resources. See: Activating and deactivating resources. • deactivate : Deactivate all matched resources. See: Activating and deactivating resources. • delete : Delete all matched resources. See: Deleting resources. -k --kind <strings> kinds array of string Include in response only resources whose kind is one of these values. -n --name <strings> names array of string Include in response only resources whose name matches one of these values. Values for this option may include the * wildcard, which matches any string in a name. For example: demo_*. -i --id <strings> ids array of string Include in response only resources whose id is one of these values. <file name> (or - to read from stdin) Accepts multiple. resources array of objects, each with kind and name string fields Include in response only resources whose kind and name equal those in one of these values. Sort response resources to match. Files specified must be in apply format. The kind and metadata.name is taken from each YAML doc in each file. --active <string> value: true or false active boolean When present, include in response only Connection and Pipeline resources that are: When true: active. When false: not active. --stable <string> value: true or false stable boolean When present, include in response only Connection and Pipeline resources that are: When true: stable. When false: not stable. --export or --no-status no_status boolean (Default: false) Return resources without status section, suitable for export for source control. --no-spec no_spec boolean (Default: false) Return resources without spec section. -M --metadata-only Combines --no-spec, --no-status, --operation=list. Only kind and the metadata section are retained in the response resources. Useful for a concise look at which resources are selected by a set of query options (specifier). --keep-ids keep_ids boolean (Default: false) Return resources with IDs in metadata and spec sections. --include-immutables include_immutables boolean (Default: false) Include in the response built-in immutable resources, such as _events and _metrics Streams. Important: These resources can’t be apply'd (or otherwise changed). Not appropriate for export to source control or apply. spec_versions object: • key: kind value • value: spec_version value (per-kind) Select the spec_version for one or more kinds. Allows your custom tooling to choose the contract (object shape) for spec sections of resource definition YAML docs in the query response. Note that spec shapes won’t be compatible across spec_versions, and spec_version is per-kind. The default spec_version is the latest for each kind. -S --output-specifier Don’t query. Output (computed) query specifier YAML instead. --stabilize Wait (poll) until all matched resources are stable. May be used without an operation, or with either activate or deactivate. See: Waiting for resources to stabilize. --timeout <string> Value: a duration Fail --stabilize process after given timeout duration. Duration string is specified as a sequence of numbers, each with a unit suffix (h, m, s), for example: 90s, 1h30m. Example usage Fetch all user resources in your Decodable account: decodable query Copy resources from one Decodable account (per CLI profile) to another by piping the output from query as input to apply (using - to denote reading from stdin): decodable -p account1 query | \ decodable -p account2 apply - Result modifier examples Export resources without the status section. This is useful for when you want to commit the resource definitions to source control (like git): decodable query --export You can use this to write it directly to a file, by redirecting stdout: decodable query --export > resources.yaml Note that --export is a synonym for --no-status. Get just the (runtime) status for resources (that’s, don’t return the full resource configuration spec): decodable query --no-spec Get just the resource kind and metadata (name, description) (excluding spec or status): decodable query --metadata-only Filter examples If you want to filter the results by conditions other than those available, you can do this with YAML post-processing. Names matching one or more * wildcard patterns, such as a prefix: decodable query --name 'demo_*' --name '*_test_*' By combining the two strings into a comma-separated list this can also be written as: decodable query --name 'demo_*,*_test_*' Specific kinds of resources: decodable query --kind 'connection,pipeline' A specific set of IDs (across kinds): decodable query --id '83ed9491,ca64352b,044b98ff' All resources defined in one or more apply-format YAML files. This is useful for checking the status of apply'd resources. decodable query resources1.yaml resources2.yaml Combining filters Each filter is optional, but all given filters must match. Here is an example that will fetch all connection and pipeline resources whose names start with demo_, but only with the given IDs, and only when appearing in resource.yaml. Example query usage that combines filters decodable query \ --kind 'connection,pipeline' \ --name 'demo_*' \ --id '83ed9491,ca64352b,044b98ff' \ resource.yaml YAML post-processing Since query returns valid YAML to stdout, you can pipe this into any YAML command-line tool for further processing. The following examples use the yq tool. Show the query YAML output with colored syntax highlighting: decodable query | yq Count resources by kind: decodable query -M \ | yq eval-all ' [.] | group_by(.kind) | map({"kind": .[0].kind, "count": length})' Count connection resources by connector: decodable query -k connection -X list \ | yq eval-all ' [.] | group_by(.spec.connector) | map({"connector": .[0].spec.connector, "count": length})' Change a word prefix in any string value to a new prefix: from=staging_ to=production_ decodable query --export \ | yq ' (.. | select(tag == "!!str")) |= sub("\b'${from}'", "'${to}'")' Filter the list of connections for just those using a particular connector, such as iceberg: decodable query -k connection | \ yq eval-all 'select(.spec.connector=="iceberg")' You can extend the above example to parameterize it: connector=iceberg decodable query -k connection | \ yq eval-all 'select(.spec.connector=="'${connector}'")' Query specifier The query command will build up a query specifier based on the optional arguments provided. You can also provide the base specifier directly as YAML, which can be useful for scripts and tools. Any additional query CLI options are added to the given YAML specifier. The specifier is an object whose fields are shown in the options table above in the "specifier field" column. You can provide the specifier as inline YAML: decodable query -y '{kinds: [connection,pipeline]}' or as a YAML file: decodable query -f specifier.yaml Example query specifier YAML kinds: - connection - pipeline names: - demo_* - "*_tmp" ids: [83ed9491,ca64352b,044b98ff] resources: - kind: connection name: demo_1_input - kind: pipeline name: demo_1_processor no_status: true To generate a query specifier file, run query with the options required, and the --output-specifier (or -S) option. Instead of running the query, it will return the computed query specifier. For example: decodable query --output-specifier --kind pipeline --name name1,name2 outputs: kinds: - pipeline names: - name1 - name2