Overview

Decodable supports reading data into a stream through a REST API. This is enabled by creating a connection using the rest connector type.

Setup Decodable REST Connection

Create a Stream

decodable stream create --name rest_stream      \
  --description "input stream for REST events"  \
  --field value=string

Create a Connection

decodable connection create --name rest_conn    \
--stream-id <streamId>                          \
--connector rest                                \
--type source

Activate the Connection

decodable connection activate <connectionId>

The connection information will be shown in your console, e.g.

rest_conn
  id                       <connectionId>
  description              
  connector                rest
  type                     source
  stream id                <streamId>
  schema
    0  value                 string
  properties
    endpoint                 /v1alpha2/connections/<connectionId>/events
  target state             RUNNING
  actual state             RUNNING
  create time              2022-01-07T00:20:38Z
  update time              2022-01-07T00:20:38Z

Take a note of the endpoint property listed, as that is the URL path to send data to the REST endpoint.

Using Decodable REST Endpoint

Payload Format

The REST connector accepts JSON in the following format:

{
  "events": [
    {
      "<fieldName>": <fieldValue>
    }
    ...
  ]
}

Using the schema defined in the above example, a valid payload would look like the following.

{
  "events": [
    {
      "value": "foo"
    },
    {
      "value": "bar"
    },
    {
      "value": "baz"
    }
  ]
}

For testing, it may be useful to save the JSON blob above to a file, e.g. payload.json

Authentication

The REST connector uses the access_token located in ~/.decodable/auth, which will be present after successfully logging in with the CLI.

For testing, you can copy this value (it's a base64-encoded JWT) and save it to an environment variable in your console, e.g.

export DECODABLE_TOKEN="<access_token value>"

Endpoint URL

The full REST endpoint URL is composed of two parts, the base-url and the endpoint.

The base-url is located in your ~/.decodable/config file. It will look like:

https://<account-name>.api.decodable.co

The endpoint is shown after an activate or get on the REST connection. It will look like:

/v1alpha2/connections/<connectionId>/events

As an example, for an account-name of test-account and a connectionId of a1b2c3d4, the full REST URL will be:

https://test-account.api.decodable.co/v1alpha2/connections/a1b2c3d4/events

Sending Events

Using the example URL above, and assuming the JSON payload is saved to /tmp/payload.json with the access_token available in the DECODABLE_TOKEN environment variable, you can send events to the REST API using:

curl -X POST https://test-account.api.decodable.co/v1alpha2/connections/a1b2c3d4/events \
  -d @/tmp/payload.json                                                                 \
  -H "Content-Type: application/json"                                                   \
  -H "Accept: application/json"                                                         \
  -H "Authorization: Bearer ${DECODABLE_TOKEN}"

You should see the following returned:

{"count":3}

Previewing Your Data

Given the stream created above with the name rest_stream, you can run the following to preview your event data as it arrives:

decodable pipeline preview "select * from rest_stream"

Submitting query... done! (took 5.68s)
Waiting for results...
{"value":"foo"}
{"value":"bar"}
{"value":"baz"}

Note that after Waiting for results... is shown, you may need to wait up to 30s before new REST messages will appear.