One of the most customizable ways to stream data into a Decodable pipeline is via the REST connector. REST (Representational State Transfer) refers to a group of software architecture design constraints that bring about efficient, reliable and scalable distributed systems. A system is called RESTful when it adheres to those constraints. The basic idea of REST is that a resource, e.g. a document, event, or record, is transferred with its state and relationships via well-defined, standardized operations and formats.
Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. REST connectors can only be used in the
To create and configure a connector for REST, sign in to the Decodable Web Console, navigate to the Connections tab, click on
New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.
The connector type will default to
source, since that is the only option for REST connectors.
Make a note of the endpoint URL that is shown. This is where your application or service will need to send HTTP POST requests to in order to be received by the REST connector. The complete URL will be in the form
https://<your_account>.api.decodable.co/v1alpha2/connections/<connection_id>/events. Once the connector has been created, you can find the connection id by double-clicking on the name of the connector. The id is shown directly below the connector name.
The value format will default to
JSON, since that is the only option for REST connectors.
Select a policy for handling record parse errors, which can be either
IGNORE. When set to FAIL, no events from a batch will be written if there are any validation errors. When set to
IGNORE, all validated events will be written, while invalid records will be ignored without stopping the connection or causing any errors/warnings. A total successful
countwill be returned in the response body.
|Delivery guarantee||at least once|
Decodable can receive data via a REST API. This is enabled by creating a connection using the
rest connector which gives you a dedicated endpoint to which you can post arrays of records. The REST connector has no connector-specific properties and is available for use once it is attached to a stream and activated.
For information about using the API request, response, headers, and other important details, see Send events to a REST connection.
The REST connector requires a valid OAuth2 token just like the other APIs.
For testing, you can use the access token used by the CLI (available in the
access_token field of
login - see logging in) by putting it in an environment variable in your shell:
export DECODABLE_TOKEN="<access_token value>"
For production applications, you must retrieve an OAuth2 token directly via the standard APIs. This will allow you to request the appropriate scopes as well as receive a refresh token so you can renew your auth token on an ongoing basis.
The REST endpoint is part of the standard Decodable REST API rooted at
https://<account-name>.api.decodable.co/. Once the connection is activated, you can use use the connection ID to post records to:
You can also find the endpoint in the output of the
decodable connection activate and
decodable connection get commands.
This API endpoint only works with REST connections, although it's possible to create REST connections for any stream.
As an example, for an
test-account and a
a1b2c3d4, the full REST URL will be:
An HTTP response of 202 guarantees that all records in the POST have been successfully written to the stream to which the connection is attached. Records are written atomically to the stream. That is, either all of the records in the POST body will be written or none of them will be written. Note that this behavior can be overridden
While unlikely, it is possible that records may be written to the stream successfully but a subsequent error (e.g. a network partition, or failure of the client or the server) occurs before the API is able to respond to the client. In this case, the client will have no choice but to retry resulting in duplicate records.
Records are guaranteed to be written in the order they appear in the
events element of the POST body.
|optional||Specify the timestamp format for |
|optional||Specify an error handling policy. Defaults to |
Updated about 1 month ago