Query existing resources

The decodable query command fetches resources from your Decodable account. It supports:

Configurable resource filters and result modifiers.
Resource export for source control.
Resource deletion.

See Declarative Resource Management overview if you are new to declarative.

With no arguments, query fetches all user resources in your Decodable account:

decodable query

See query examples and options reference.

Query output

The query command outputs a series of resource definitions, each as a YAML doc that extends the apply input format with a status section. The status contains non-writable fields managed by Decodable for the resource.

Example query output

---
kind: pipeline
metadata:
    name: example_pipe
    description: Perform example processing
spec_version: v1
spec:
    sql: |
        INSERT INTO example_out
        SELECT    lower(s)
        FROM      example_in
status:
    version: 1
    is_latest: true
    type: SQL
    latest_version: 1
    target_state: RUNNING
    actual_state: RUNNING
    requested_tasks: 2
    actual_tasks: 2
    create_time: 2024-04-26T17:19:35.311+00:00

The status section is ignored by the apply command, so the output from query can serve directly as input to decodable apply.

Filtering resources

The query command can filter resources (server-side) in four distinct ways, which may be combined.

For example, to fetch all resources whose names start with demo_:

decodable query --name 'demo_*'

You can filter resources by:

kind
name (with * wildcard)
id
Per-resource kind and name, via input resource files

See:

Complete query filter examples.
Options reference.

Exporting existing resources to source control

Use the query command to export your Decodable resources for storing in a source control system like git.

Maintaining resources via files in source control serves as the basis for "infrastructure-as-code" development practices. The apply command may be used to create or update resources to match the resource file(s) in source control. This can be automated via CI/CD.

Always use the --export (or --no-status) option when exporting for source control. This option suppresses the status section in query result resources. Although apply ignores the status section, status is not appropriate for resource definitions in source control.

Never use --keep-ids when exporting for source control: IDs cannot be replicated to new resources (after a resource deletion or in a different account, for example), and so break repeatability.

For example:

decodable query --export > decodable_resources.yaml
git add decodable_resources.yaml
git commit -m "IaC FTW!" # obviously you use better commit messages than this ;)

# ... elsewhere later ...
git pull
decodable apply decodable_resources.yaml

Deleting resources

Use the query command with the delete operation to remove resources. This is compatible with any combination of query filters.

For example, to delete all resources whose names start with demo_:

decodable query --name 'demo_*' --operation delete

Deleting safely

The delete operation is a powerful tool. Deletion is permanent.

To use delete safely, try your query filters first without the delete operation, inspect the output, then add the --operation delete (or -X delete) option.

Even safer: Redirect the fetch output to a file, inspect the file, then use that file as input for the delete. This ensures that only the resources in that file will be deleted, not any additional resources that may have been added after the query fetch that match the same filters. For example:

decodable query --name 'demo_*' > to_delete.yaml
# ... review, inspect ...
decodable query -X delete to_delete.yaml

Understanding `delete`

Like the apply command, the query delete operation is atomic: it either deletes all matched resources successfully or fails with an error, deleting nothing.

The delete operation returns apply-style resource status format YAML, rather than resource definition format YAML returned by non-mutating (fetching) query operations.

Currently, delete will fail if any matched runnable resources are RUNNING. This applies to connections and pipelines. To delete a resource set that includes such resources, deactivate/stop them first.

The delete operation requires at least one filter, for safety. If you really want to delete all resources in your Decodable account, use: --name '*'.

Deleting resources from `apply`

The query delete operation also removes a set of resources created or modified by the apply command:

decodable apply resources.yaml
# ... use these resources ...
decodable query resources.yaml -X delete

Result order will match that of the resources in the input file(s).

Note that any resources specified in file(s) must (still) exist to be deleted. Specified resources that don’t exist are simply ignored.

Arguments guide and options reference

The query command (and corresponding API endpoint) supports many options, all optional. Most may be specified via command line (CLI) options, some of which allow a short form. All may be given via query specifier YAML.

Each query is controlled with a query specifier object. The query command builds the specifier for you based on the given command line arguments. You can also provide the query specifier directly as YAML.

Resource definition files as filter

In addition to command line options, query accepts optional input file names. These must be YAML files with resource definitions, in apply input / query output format. If a file name is -, standard input (stdin) is used.

When file names are provided, results are:

filtered to include only resources in the specified files (by kind and name);
sorted to match the order of resources in the specified files;

This can be useful after an apply to check the status of applied resources:

decodable apply resources.yaml
decodable query resources.yaml

file names and other filter options may be used together. If other filter options are given, all filters much match: only resources defined in the given files and matching all other given filters will be included in the results.

Shell globbing (with *) can be an effective way to provide a set of input files. For example, decodable query resources/yaml.* will query the state of all resources in all YAML files in the resources directory.

Note that the -f/--specifier-file option also takes a file name, but this is in a different format and serves a different purpose. See table below.

Options reference

CLI options that take <strings> allow multiple values, either by repeating the option or using a comma-separated (,) list.

short long option specifier field description

short	long option	specifier field	description
`-y`	`--specifier-yaml <yaml>`		Provide (base) query specifier as inline YAML.
`-f`	`--specifier-file <file name>`		Provide (base) query specifier as a YAML file. If the file name is `-`, standard input (`stdin`) is used. Note that this is different from providing a file name on its own, not using this option.
`-X`	`--operation <string>`	`operation` string	Query operation: one of: • `get` (default): Include associated values in each resource: schema, properties, etc. • `list` : Include only scalar values. This can be faster in some cases. • `delete` : Delete all matched resources. See: Deleting resources.
`-k`	`--kind <strings>`	`kinds` array of string	Include in response only resources whose `kind` is one of these values.
`-n`	`--name <strings>`	`names` array of string	Include in response only resources whose `name` matches one of these values. Values for this option may include the `` wildcard, which matches any string in a name. For example: `demo_`.
`-i`	`--id <strings>`	`ids` array of string	Include in response only resources whose `id` is one of these values.
	`<file name>` (or `-` to read from `stdin`) Accepts multiple.	`resources` array of objects, each with `kind` and `name` string fields	Include in response only resources whose `kind` and `name` equal those in one of these values. Sort response resources to match. File(s) specified must be in `apply` format. The `kind` and `metadata.name` is taken from each YAML doc in each file.
	`--export` or `--no-status`	`no_status` boolean (Default: `false`)	Return resources without `status` section, suitable for export for source control.
	`--no-spec`	`no_spec` boolean (Default: `false`)	Return resources without `spec` section.
`-M`	`--metadata-only`		Combines `--no-spec`, `--no-status`, `--operation=list`. Only `kind` and the `metadata` section are retained in the response resources. Useful for a concise look at which resources are selected by a set of `query` options (specifier).
	`--keep-ids`	`keep_ids` boolean (Default: `false`)	Return resources with ids in spec section.
	`--include-immutables`	`include_immutables` boolean (Default: `false`)	Include in the response built-in immutable resources, such as `_events` and `_metrics` Streams. Important: These resources cannot be `apply`'d (or otherwise changed). Not appropriate for export to source control or `apply`.
		`spec_versions` object: • key: `kind` value • value: `spec_version` value (per-`kind`)	Select the `spec_version` for one or more `kind`s. Allows your custom tooling to choose the contract (object shape) for `spec` sections of resource definition YAML docs in the `query` response. Note that `spec` shapes will not be compatible across `spec_version`s, and `spec_version` is per-`kind`. The default `spec_version` is the latest for each `kind`.
`-S`	`--output-specifier`		Don’t query. Output (computed) query specifier YAML instead.

-y

--specifier-yaml <yaml>

Provide (base) query specifier as inline YAML.

-f

--specifier-file <file name>

Provide (base) query specifier as a YAML file.

If the file name is -, standard input (stdin) is used.

Note that this is different from providing a file name on its own, not using this option.

-X

--operation <string>

operation

string

Query operation: one of:

• get (default): Include associated values in each resource: schema, properties, etc.

• list : Include only scalar values. This can be faster in some cases.

• delete : Delete all matched resources. See: Deleting resources.

-k

--kind <strings>

kinds

array of string

Include in response only resources whose kind is one of these values.

-n

--name <strings>

names

array of string

Include in response only resources whose name matches one of these values.

Values for this option may include the * wildcard, which matches any string in a name. For example: demo_*.

-i

--id <strings>

ids

array of string

Include in response only resources whose id is one of these values.

<file name> (or - to read from stdin)

Accepts multiple.

resources

array of objects, each with kind and name string fields

Include in response only resources whose kind and name equal those in one of these values. Sort response resources to match.

File(s) specified must be in apply format. The kind and metadata.name is taken from each YAML doc in each file.

--export

--no-status

no_status

boolean (Default: false)

Return resources without status section, suitable for export for source control.

--no-spec

no_spec

boolean (Default: false)

Return resources without spec section.

-M

--metadata-only

Combines --no-spec, --no-status, --operation=list.

Only kind and the metadata section are retained in the response resources.

Useful for a concise look at which resources are selected by a set of query options (specifier).

--keep-ids

keep_ids

boolean (Default: false)

Return resources with ids in spec section.

--include-immutables

include_immutables

boolean (Default: false)

Include in the response built-in immutable resources, such as _events and _metrics Streams.

Important: These resources cannot be apply'd (or otherwise changed). Not appropriate for export to source control or apply.

spec_versions

object:

• key: kind value

• value: spec_version value (per-kind)

Select the spec_version for one or more kinds.

Allows your custom tooling to choose the contract (object shape) for spec sections of resource definition YAML docs in the query response. Note that spec shapes will not be compatible across spec_versions, and spec_version is per-kind.

The default spec_version is the latest for each kind.

-S

--output-specifier

Don’t query. Output (computed) query specifier YAML instead.

Example usage

Fetch all user resources in your Decodable account:
```
decodable query
```
Copy resources from one Decodable account (per CLI profile) to another by piping the output from query as input to apply (using - to denote reading from stdin):
```
decodable -p account1 query   | \
decodable -p account2 apply -
```

Result modifier examples

Export resources without the status section.

This is useful for when you want to commit the resource definitions to source control (like git):
```
decodable query --export
```
You can use this to write it directly to a file, by redirecting stdout:
```
decodable query --export > resources.yaml
```
Note that --export is a synonym for --no-status.
Get just the (runtime) status for resources (that is, don’t return the full resource configuration spec):
```
decodable query --no-spec
```
Get just the resource kind and metadata (name, description) (excluding spec or status):
```
decodable query --metadata-only
```

Filter examples

If you want to filter the results by conditions other than those available, you can easily do this with YAML post-processing.

Names matching one or more * wildcard patterns, such as a prefix:
```
decodable query --name 'demo_*' --name '*_test_*'
```
By combining the two strings into a comma-separated list this can also be written as:
```
decodable query --name 'demo_*,*_test_*'
```

Specific kinds of resources:

decodable query --kind 'connection,pipeline'

A specific set of IDs (across kinds):

decodable query --id '83ed9491,ca64352b,044b98ff'

All resources defined in one or more apply-format YAML files. This is useful for checking the status of apply'd resources.
```
decodable query resources1.yaml resources2.yaml
```

Combining filters

Each filter is optional, but all given filters must match.

Here is an example that will fetch all connection and pipeline resources whose names start with demo_, but only with the given IDs, and only when appearing in resource.yaml.

Example query usage that combines filters

decodable query                        \
 --kind 'connection,pipeline'          \
 --name 'demo_*'                       \
 --id   '83ed9491,ca64352b,044b98ff'   \
 resource.yaml

YAML post-processing

Since query returns valid YAML to stdout, you can pipe this into any YAML command-line tool for further processing.

The following examples use the yq tool.

Show the query YAML output with colored syntax highlighting:
```
decodable query | yq
```

Count resources by kind:

decodable query -M \
| yq eval-all '
  [.]
  | group_by(.kind)
  | map({"kind": .[0].kind, "count": length})'

Count connection resources by connector:

decodable query -k connection -X list \
| yq eval-all '
  [.]
  | group_by(.spec.connector)
  | map({"connector": .[0].spec.connector, "count": length})'

Change a word prefix in any string value to a new prefix:

from=staging_
to=production_
decodable query --export \
| yq '
  (.. | select(tag == "!!str"))
  |= sub("\b'${from}'", "'${to}'")'

Filter the list of connections for just those using a particular connector, such as iceberg:
```
decodable query -k connection | \
  yq eval-all 'select(.spec.connector=="iceberg")'
```

You can extend the above example to parameterize it:

connector=iceberg
decodable query -k connection | \
  yq eval-all 'select(.spec.connector=="'${connector}'")'

Query specifier

The query command will build up a query specifier based on the optional arguments provided.

You can also provide the base specifier directly as YAML, which can be useful for scripts and tools. Any additional query CLI options are added to the given YAML specifier.

The specifier is an object whose fields are shown in the options table above in the "specifier field" column.

You can provide the specifier as inline YAML:

decodable query -y '{kinds: [connection,pipeline]}'

or as a YAML file:

decodable query -f specifier.yaml

Example query specifier YAML

kinds:
  - connection
  - pipeline
names:
  - demo_*
  - "*_tmp"
ids: [83ed9491,ca64352b,044b98ff]
resources:
  - kind: connection
    name: demo_1_input
  - kind: pipeline
    name: demo_1_processor
no_status: true

To generate a query specifier file, run query with the options required, and the --output-specifier (or -S) option. Instead of running the query, it will return the computed query specifier. For example:

decodable query --output-specifier  --kind pipeline --name name1,name2

outputs:

kinds:
    - pipeline
names:
    - name1
    - name2

Query existing resources

Query output

Filtering resources

Exporting existing resources to source control

Deleting resources

Deleting safely

Understanding delete

Deleting resources from apply

Arguments guide and options reference

Resource definition files as filter

Options reference

Example usage

Result modifier examples

Filter examples

Combining filters

YAML post-processing

Query specifier

Example query specifier YAML

Understanding `delete`

Deleting resources from `apply`