One of the most customizable ways to stream data into a Decodable pipeline is via the REST connector. REST (Representational State Transfer) refers to a group of software architecture design constraints that bring about efficient, reliable and scalable distributed systems. A system is called RESTful when it adheres to those constraints. The basic idea of REST is that a resource, e.g. a document, event, or record, is transferred with its state and relationships via well-defined, standardized operations and formats.

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. REST connectors can only be used in the source role.

Configure As A Source

To create and configure a connector for REST, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

1. The connector type will default to source, since that is the only option for REST connectors.

2. Make a note of the endpoint URL that is shown. This is where your application or service will need to send HTTP POST requests to in order to be received by the REST connector. The complete URL will be in the form https://<your_account>.api.decodable.co/v1alpha2/connections/<connection_id>/events. Once the connector has been created, you can find the connection id by double-clicking on the name of the connector. The id is shown directly below the connector name.

3. The value format will default to JSON, since that is the only option for REST connectors.

4. Select a policy for handling record parse errors, which can be either FAIL or IGNORE. When set to FAIL, no events from a batch will be written if there are any validation errors. When set to IGNORE, all validated events will be written, while invalid records will be ignored without stopping the connection or causing any errors/warnings. A total successful count will be returned in the response body.

For more detailed information about creating and configuring a REST API, see the REST connection API reference and the REST connector How To guide.

Reference

Decodable can receive data via a REST API. This is enabled by creating a connection using the rest connector which gives you a dedicated endpoint to which you can post arrays of records. The REST connector has no connector-specific properties and is available for use once it is attached to a stream and activated.

For information about using the API request, response, headers, and other important details, see Send events to a REST connection.

Authentication

The REST connector requires a valid OAuth2 token just like the other APIs.

For testing, you can use the access token used by the CLI (available in the access_token field of ~/.decodable/auth after login - see logging in) by putting it in an environment variable in your shell:

export DECODABLE_TOKEN="<access_token value>"

For production applications, you must retrieve an OAuth2 token directly via the standard APIs. This will allow you to request the appropriate scopes as well as receive a refresh token so you can renew your auth token on an ongoing basis.

Endpoint URL

The REST endpoint is part of the standard Decodable REST API rooted at https://<account-name>.api.decodable.co/. Once the connection is activated, you can use use the connection ID to post records to:

/v1alpha2/connections/<connection-id>/events.

You can also find the endpoint in the output of the decodable connection activate and decodable connection get commands.

This API endpoint only works with REST connections, although it's possible to create REST connections for any stream.

As an example, for an account-name of test-account and a connection-id of a1b2c3d4, the full REST URL will be:

https://test-account.api.decodable.co/v1alpha2/connections/a1b2c3d4/events

Guarantees

An HTTP response of 202 guarantees that all records in the POST have been successfully written to the stream to which the connection is attached. Records are written atomically to the stream. That is, either all of the records in the POST body will be written or none of them will be written. Note that this behavior can be overridden

While unlikely, it is possible that records may be written to the stream successfully but a subsequent error (e.g. a network partition, or failure of the client or the server) occurs before the API is able to respond to the client. In this case, the client will have no choice but to retry resulting in duplicate records.

Records are guaranteed to be written in the order they appear in the events element of the POST body.

Properties