Imply Polaris sink connector Use the Imply Polaris Connector to send data from Decodable to a Polaris table. Features Delivery guarantee At least once Prerequisites: Prepare Polaris for connections Before you send data from Decodable into Imply Polaris, do the following in your Polaris account to make sure that you are able to create a connection to it. Create a table in Polaris. See Create a table by API for instructions on how to create a table. Create a Polaris push_streaming connection and a streaming job to your Polaris table. The input schema specified in the streaming job and the schema of the Decodable stream that is sending data to Polaris must match. Otherwise, your data may not appear in the Polaris table. See Create a table by API for instructions on how to create Polaris connections and jobs. Make sure that an event timestamp field exists in the Decodable stream schema. You must map the timestamp field in Decodable to the timestamp field in the Polaris streaming job. Steps: Create a connection with the Imply Polaris connector If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is polaris. From the Connections page, select the Imply Polaris Connector and complete the following fields. Property Name Description Connection Name The name of the push_streaming connection associated with the Polaris table you want to send data to. If you want to send data to a Polaris v1 table, then this is the same as the Table ID. Organization Name The name of the organization containing the Polaris table you want to send data to. Client ID The OAuth client ID obtained from Polaris. Client Secret The client secret associated with the OAuth client ID. Cloud Provider The cloud service provider for your Polaris project. Currently, only aws is supported. Region The region that the Polaris project is in. The following regions are supported: us-east-1 eu-central-1 Select which stream contains the records you want to send to Polaris. Then, select Next. Give the newly created connection a Name and Description and select Save. Start this connection to begin sending data from Decodable to Polaris. Example: Use the REST API to send data from Decodable to Imply Polaris Let’s walk through an example of the four steps above. We will use the Polaris REST API to illustrate connection and job creation in Polaris. We will also use the Decodable REST API to illustrate creating the sink connection to Imply Polaris. (NOTE: this workflow can also be done via the UI/CLI of both services as well). In this example, assume that we’ve already created a Polaris table called http_requests with the following schema: | | | | __time | method | original_path | response_code | bytes_rcvd | | | | In addition, we’ve also already created a Decodable stream called http_requests_stream with the following schema: | | | | timestamp | method | original_path | response_code | bytes_rcvd | | | | Notice in this example that the timestamp field is called _time in the Polaris table and timestamp in the Decodable stream. We will create a mapping between these two fields in the streaming job as part of step 1. Create a push_streaming connection and a streaming job using the Polaris REST API. curl --location --request POST 'https://ORGANIZATION_NAME.api.imply.io/v2/connections' \ --header 'Authorization: Bearer $IMPLY_TOKEN' \ --header 'Content-Type: application/json' \ --data-raw '{ "type": "push_streaming", "name": "demo_connection" }' curl --location --request POST 'https://ORGANIZATION_NAME.api.imply.io/v2/jobs' \ --header "Authorization: Bearer $IMPLY_TOKEN" \ --header 'Content-Type: application/json' \ --data-raw '{ "type": "streaming", "target": { "type": "table", "tableName": "http_requests" }, "source": { "type": "connection", "connectionName": "demo_connection", "inputSchema": [ { "name": "timestamp", "dataType": "string" }, { "name": "method", "dataType": "string" }, { "name": "original_path", "dataType": "string" }, { "name": "response_code", "dataType": "long" }, { "name": "bytes_rcvd", "dataType": "long" } ], "formatSettings": { "format": "nd-json" } }, "mappings": [ { "columnName": "__time", "expression": "TIME_PARSE(\"timestamp\")" }, { "columnName": "method", "expression": "\"method\"" }, { "columnName": "original_path", "expression": "\"original_path\"" }, { "columnName": "response_code", "expression": "\"response_code\"" }, { "columnName": "bytes_rcvd", "expression": "\"bytes_rcvd\"" } ] }' Create a connection to Imply Polaris using the Decodable REST API. curl --request POST \ --url https://<your_decodable_organization>.api.decodable.co/v1alpha2/connections \ --header 'Accept: application/json' \ --header 'Authorization: Bearer $DECODABLE_AUTH_TOKEN' \ --header 'Content-Type: application/json' \ --data ' { "connector": "polaris", "properties": { "polaris.client-id": "<your_client_ID>", "polaris.connection-name": "<your_connection_name>", "polaris.org-name": "<your_org_name>", "polaris.client-secret": "<your_client_secret>", "polaris.cloud": "<cloud_provider">, "polaris.region": "<region>" }, "name": "imply_polaris_demo", "description": "Send data to Imply Polaris from Decodable", "type": "sink", "stream_id": "<http_requests_stream ID>" } ' Start the imply_polaris_demo connection from the REST API/UI/CLI and sit back, pet your cat, take a walk, or start working with your data in Polaris!