One of the most customizable ways to stream data into a Decodable pipeline is via the REST connector. REST (Representational State Transfer) refers to a group of software architecture design constraints that bring about efficient, reliable and scalable distributed systems. A system is called RESTful when it adheres to those constraints. The basic idea of REST is that a resource, e.g. a document, event, or record, is transferred with its state and relationships via well-defined, standardized operations and formats.

Overview


Connector name	`rest`
Types	`source`
Delivery guarantee	at least once

Configure As A Source

Follow these steps to get data into Decodable using the REST Connector.

From the Connections page, select the REST Connector and complete the following fields.

UI Field	Property Name in the Decodable CLI	Description
Endpoint	`endpoint`	The location where your application or service will need to send HTTP POST requests to. The complete URL will be in the format `https://<your_account>.api.decodable.co/v1alpha2/connections/<connection_id>/events`. Once the connection has been created, you can find the connection id by double-clicking on the name of the connector. The id is shown directly below the connector name.
Value Format	`format`	Must be set to JSON.
Parse error policy	`parse-error-policy`	Optional. Select the error handling policy. Must be one of the following: - `FAIL`: When set to FAIL, Decodable stops the connection if any validation errors are encountered in the incoming data stream. - `IGNORE`: When set to IGNORE, Decodable ignores all invalid records. All validated records are sent. With this policy, the connection is not stopped nor will any errors or warnings be shown if there is an invalid record. A total successful `count` will be returned int he response body. Defaults to `FAIL`.

Select the stream that you’d like to connect to this connector. This will be the stream that receives events using REST. Then, select Next.
Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Import Schema if you want to paste the schema in the form of an Avro or JSON array.
1. The stream's schema must match the schema of the data that you plan on sending through this connection.
2. For more information about creating a stream or defining the stream schema, see Create a Stream.
Select Next when you are finished providing defining the connection’s schema.
Give the newly created connection a Name and Description and select Save.

You are now ready to send data to Decodable using the REST Connector! For detailed information about sending data, see the following topics:

For information about using the API request, response, headers, and other important details, see Send events to a REST connection.
For a guided walkthrough of sending data to the Decodable REST endpoint, see the REST connector How To guide.

Reference

Authentication

The REST connector requires a valid OAuth2 token just like the other APIs.

For testing, you can use the access token used by the CLI (available in the access_token field of ~/.decodable/auth after login - see logging in) by putting it in an environment variable in your shell:

export DECODABLE_TOKEN="<access_token value>"

For production applications, you must retrieve an OAuth2 token directly via the standard APIs. This will allow you to request the appropriate scopes as well as receive a refresh token so you can renew your auth token on an ongoing basis.

Endpoint URL

The REST endpoint is part of the standard Decodable REST API rooted at https://<account-name>.api.decodable.co/. Once the connection is activated, you can use use the connection ID to post records to:

/v1alpha2/connections/<connection-id>/events.

You can also find the endpoint in the output of the decodable connection activate and decodable connection get commands.

This API endpoint only works with REST connections, although it's possible to create REST connections for any stream.

As an example, for an account-name of test-account and a connection-id of a1b2c3d4, the full REST URL will be:

https://test-account.api.decodable.co/v1alpha2/connections/a1b2c3d4/events

Guarantees

An HTTP response of 202 guarantees that all records in the POST have been successfully written to the stream to which the connection is attached. Records are written atomically to the stream. That is, either all of the records in the POST body will be written or none of them will be written. Note that this behavior can be overridden.

While unlikely, it is possible that records may be written to the stream successfully but a subsequent error (e.g. a network partition, or failure of the client or the server) occurs before the API is able to respond to the client. In this case, the client will have no choice but to retry resulting in duplicate records.

Records are guaranteed to be written in the order they appear in the events element of the POST body.