Decodable APIs Decodable has a control plane and data plane separated architecture. What this means is that Decodable services are divided into two categories: Control plane services and Data plane services. Each of these two services have their own API endpoints. See the Control Plane and Data Plane API Reference for more information, including: which endpoints are in the control plane and which endpoints are in the data plane, what arguments each endpoint accepts, and more. The Pipeline SDK is available for use with pipelines built using your own Apache Flink jobs. Control Plane The control plane is responsible for managing the overall configuration of a Decodable account. Specifically, it’s responsible for account management, account identity and authorization, the data catalog, SQL parsing and analysis, access control policy definitions, and other "metadata" about the account. Services in the control plane allow Decodable to control connection and pipeline lifecycle, manage dependencies, and understand the status of processing that occurs in the data plane. Data Plane While the control plane handles the high-level decision-making and management aspects of Decodable, the data plane is where Decodable actually processes data. Credentials are also stored in the data plane, and user data never leaves the data plane. How the client authenticates to the Data Plane Because the data plane is where your user data and credentials reside and the control plane is where access control is managed, every request to a data plane API endpoint must include a data plane request token that’s signed by the control plane API service. To obtain a data plane request token that you can use, you must first send a request to the corresponding control plane’s token endpoint. There are separate control plane token endpoints for each data plane action, and the token returned is only authenticated for that particular action. Then, you can make a request to the data plane API using the data plane request token. The data plane request token is referred to as the "JWT token" in the API Reference. To summarize, the general steps for authenticating and making requests to the data plane are: Obtain a signed data plane request token from one of the control plane’s token endpoints. Make sure that you are using the right token endpoint for the data plane request you would like to make. For example, if you want a token to use to retrieve metrics for a connection, you must use https://<account>.api.decodable.co/v1alpha2/connections/<connection_id>/metrics/token endpoint. See the Control Plane and Data Plane API Reference for more information. Make the desired data plane request using the data plane request token to authenticate to the data plane API. See Example: Use the Decodable APIs to preview a Decodable stream for an example. High level overview of clients interact with the Control Plane and Data Plane APIs The following diagram is a high level overview of how you, the client, can interact with the Control Plane and Data Plane APIs. Example: Use the Decodable APIs to preview a Decodable Stream To illustrate the changes to the Decodable API and how you interact with them, the following example describes the new workflow for previewing a stream. Retrieve the data plane host name. The placeholders in this query are defined as follows: <account_name>: The name of your Decodable account. Make sure you are entering the actual account name, and not the display name. <access_token>: The unique token used to authenticate to Decodable. After you login with the Decodable CLI, you can find this in the auth file in the ~/.decodable/ directory. curl "https://<account_name>.api.decodable.co/v1alpha2/accounts/<account_name>" \ -H "Authorization: Bearer <access_token>" The data plane host name is returned in the response. For example: { "subscription": { "plan_id": "free", "capabilities": [] }, "deployment_type": "managed", "data_plane_hostname": "1.api.data.decodable.co", "uses_new_secrets": true } Get a data plane request token from the control plane. Make sure that you are using the right token endpoint for the data plane request you would like to make. In this case, we want to preview a stream, so we will use the https://<account_name>.decodable.co/v1alpha2/streams/<connection_id>/preview/token endpoint. The token is different from the general access token. The general access token is used to authenticate to Decodable’s control plane. The new placeholders in this query are defined as follows: <stream_id>: The ID of the stream that you want to preview. curl "https://<account_name>.decodable.co/v1alpha2/streams/<stream_id>/preview/token" \ -X 'POST' \ -H 'Authorization: Bearer <access_token>' The data plane request token is returned in the response. Some of the other /token endpoints include a data_plane_request property in the response as well. You don’t need this property value to preview a stream, but other data plane requests might require it. If the data_plane_hostname is present in the token response, it should be used as the body for the corresponding data plane request. Preview the stream. The placeholders in this query are defined as follows: <data_plane_hostname>: The data plane host name that you retrieved in Step 1. <stream_id>: The ID of the stream that you want to preview. <data_plane_request_token>: The data plane request token that you obtained in Step 2. curl 'https://<data_plane_hostname>/v1alpha2/streams/<stream_id>/preview' \ -H "Authorization: Bearer <data_plane_request_token>" The stream preview is shown in the response.