Declarative definition reference

The following are templates of different resource definitions in the apply YAML format. Use them as a reference for creating your own declarative definitions.

Connections

A template for a connection definition.

---
kind: connection
metadata:
  name: <name_your_connection>
  description: <description>
spec_version: v1
spec:
  connector: <connector_name>
  type: <type of connector>
  properties:
    <property>: <value>
    <property_2>: <value>
  stream_name: <stream>
  schema_v2:
    fields:
      - kind: <kind of field>
        name: <field name>
        type: <data type>
Field Description

spec.connector

The name of the connector. For example: mysql-cdc or s3-v2.

spec.type

The type of connector. Enter source if your connector receives data or sink if your connector sends data.

spec.properties

Each connector has its own set of properties that you can specify. To determine the property names and their valid values for a specific connector, refer to the corresponding documentation page for that connector.

Note: Secret properties, such as passwords, use the name of a secret resource, not the actual password or secret plaintext value.

spec.stream_name

The name of the stream that you want the connection to send or receive data from.

spec.schema_v2

The schema of the connection. This must match the schema of the stream that it is connected to.

Streams

A template for a stream definition.

---
kind: stream
metadata:
  name: <name_your_stream>
  description: <description>
spec_version: v1
spec:
  schema_v2:
    fields:
      - kind: <kind of field>
        name: <field name>
        type: <data type>
    constraints:  # optional
      primary_key:
        - <field name>
    watermarks:  # optional
      - name: <name>
        expression: <watermark expression>
Field Required or optional Description

spec.schema_v2

Required

The schema of the stream. This must match the schema of the connection that it is connected to.

spec.schema_v2.fields.kind

Required

One of: physical, computed, metadata.

For more information on these options, see the table in How to use the schema view.

spec.schema_v2.constraints.primary_key

Optional

Specify one or more field(s) to use as a primary key or partition key.

You must (only) specify a primary key if you are sending Change Data Capture (CDC) records. For more information on partition keys or primary keys, see the table in How to use the schema view.

spec.schema_v2.watermarks

Optional

A field that can be used to track the progression of event time in a pipeline.

Secrets

A template for a secret definition.

---
kind: secret
metadata:
  name: <name_your_secret>
  description: <description>
spec_version: v1

Note that there is no spec for secret, but spec_version: v1 is still required.

Remember that a secret resource is created without a plaintext value (the actual password, for example), which must be set after creation.

Pipelines

A template for a pipeline definition.

---
kind: pipeline
metadata:
  name: <name_your_pipeline>
  description: <description>
spec_version: v1
spec:
  sql: |
    INSERT INTO <destination stream>
    SELECT ...
    FROM <source stream>
Field Required or optional Description

spec.sql

Required

A SQL statement in the form:

INSERT INTO <destination stream> SELECT …​ FROM <source stream>.

For example:

INSERT INTO my_destination_stream SELECT lower(s) FROM my_source_stream.
Currently only SQL pipelines are supported by apply. Declarative support for Custom Pipelines is on the way.