Creating a connection

Connections in Decodable ingest data from a data source or send data out to a data sink. You can find a full list of available connectors here, or in the navigation menu on the left.

There are four ways to manage connections in Decodable: the Decodable web app, the CLI, and the API. Within the CLI you may use an imperative approach, or a declarative YAML-based declarative approach.

Web App
CLI
Declarative Resource Management
API

Navigate to the Connections page.
On the Connections page, click the "New Connection" button in the upper right corner.
Select the type (source or sink) and the external system you want to connect to from the list of available connectors.
Enter the connectivity configuration and credentials in the dialog that opens and click "Next." The following behavior depends on whether the connector supports connecting to multiple streams or not.
1. If the chosen connector supports writing to or reading from multiple streams:
  1. A dialog appears from which you can configure the mappings between external resources and Decodable streams.
  2. Select which resources should be mapped by the connection by checking the corresponding "Sync" checkboxes, and which sinks their data should be sent to.
  3. Click "Next," choose a name in the following dialog window, and click "Create" or "Create and start."
    
    For a programmatic approach to this step, see declarative resource management and the connection scan command provided by the CLI.
2. If the chosen connector only supports reading from or writing to a single stream:
  1. A dialog appears from which you can either select an existing stream or create a new stream that your connection will send data to (if its type is source) or get data from (if its type is sink). After making a choice, click "Next."
  2. In the following window you can define the schema of records flowing through this connection if you are creating a new stream. If you use an existing stream, its schema will be carried over.
  3. Click "Next," choose a name in the following dialog window, and click "Create" or "Create and start."

To use the CLI, download it and then login to your account

Create a connection using the decodable connection create command:

decodable connection create        \
      --name my-datagen-connection \
      --type source                \
      --connector datagen          \
      --prop delay=1000            \
      --prop format=json           \
      --prop data.type=envoy       \
      --field message="STRING"

For the connector value and specific configuration properties (prop) consult the connector’s documentation. The connector used in the example above is the Data Generator connector.

You should get a message confirming the connection’s creation:

Created connection my-datagen-connection (64ae6213)

The connection is created in an inactive state. To start the connection, use decodable connection activate and the id of the connection. The id of the connection is shown in the creation confirmation.
```
decodable connection activate 64ae6213
```
```
my-datagen-connection
  id                       64ae6213
[…]
  target state             RUNNING
  actual state             STARTING
```

To find out more about managing connections with the CLI consult the relevant documentation or run decodable connection --help to see the list of available subcommands.

You can use YAML to define your resources, including connections, and manage them in a declarative way. This approach integrates seamlessly with CI/CD tools and allows for management of large-scale and complex resources. To learn more about using declarative resource management consult the documentation pages.

For connectors that support multiple streams the decodable scan command is an easy way to generate YAML files. You can also use decodable query to return the definitions of existing resources in YAML as a starting point for creating new ones.

The Decodable CLI provides access to the declarative resource management commands. Before using the examples below, download the Decodable CLI and login to your account.

To create a connection using declarative resource management:

Create a YAML file with your resource definitions:

connection.yaml

---
kind: connection
metadata:
    name: my-datagen-connection
spec_version: v2
spec:
    connector: datagen
    type: source
    stream_name: my-datagen-connection_source
    schema_v2:
        fields: &envoy
            - kind: physical
              name: message
              type: STRING
        constraints: {}
    properties:
        delay: "1000"
        format: json
        data.type: envoy
    execution:
        active: true
        task_count: 1
        task_size: S
---
kind: stream
metadata:
    name: my-datagen-connection_source
spec_version: v1
spec:
    schema_v2:
        fields: *envoy

For the connector value and specific configuration properties consult the connector’s documentation. The connector used in the example above is the Data Generator connector.

For the full syntax of the YAML used, see the resource definition.

Apply this definition using decodable apply. Note that the definition includes the connection’s execution status; in this example, the connection will be both created and activated.
```
$ decodable apply connection.yaml
```
The command will return the output of the command:
```
---
kind: connection
name: my-datagen-connection
id: 670e5b5a
result: created
```

Confirm the connection’s status using decodable query:

$ decodable query --name my-datagen-connection --no-spec

You should see that it has been created and started running:

---
kind: connection
metadata:
    name: my-datagen-connection
spec_version: v2
status:
    create_time: 2024-09-26T13:16:51.244+00:00
    update_time: 2024-09-26T13:16:51.244+00:00
    last_activated_time: 2024-09-26T13:16:51.244+00:00
    execution:
        state: RUNNING

Use a POST HTTP call to the CreateConnection Decodable control plane API endpoint.

The URL of the endpoint is based on your account name:

https://<account name>.api.decodable.co/v1alpha2/connections/

Below is an example of creating a connection under an account called acme-01 and using curl to make the HTTP call.

You will need an access token in order to authenticate your REST call to the Decodable API endpoint. You can obtain this using the Decodable CLI's decodable token access command. For more details see here.

Prepare the request payload for CreateConnection based on the connector that you want to use. Consult the connector’s documentation page for details of the connector name and properties. The connector used in the example below is the Data Generator connector.
my-datagen-connection.json
```
{
  "name": "my-datagen-connection",
  "connector": "datagen",
  "type": "source",
  "schema": [
    {
      "name": "message",
      "type": "STRING"
    }
  ],
  "properties": {
    "format": "json",
    "delay": "1000",
    "data.type": "envoy"
  }
}
```

POST the payload to the CreateConnection endpoint.

The access token is obtained using command substitution. This relies on the Decodable CLI being available and already logged in on the machine making the curl call.

curl -s -XPOST "https://acme-01.api.decodable.co/v1alpha2/connections/" \
  -H "Authorization: Bearer $(decodable token access)"  \
  -H 'Content-Type: application/json' \
  -d @my-datagen-connection.json

You’ll get a response payload per the API spec including the id of connection:

{
  "id": "0abe42bf",
  "name": "my-datagen-connection",
[…]
  "target_state": "STOPPED",
  "actual_state": "STOPPED",
[…]
}

To activate the new connection use the ActivateConnection endpoint. Here is an example of activating the connection created above. We take the id 0abe42bf from the response payload.
```
curl -s -XPOST "https://acme-01.api.decodable.co/v1alpha2/connections/0abe42bf/activate" \
     -H "Authorization: Bearer $(decodable token access)"
```