Imply Polaris

Use the Imply Polaris Connector to send data from Decodable to a Polaris table.

Overview

Connector name

polaris

Type

sink

Delivery guarantee

at least once

Prerequisites: Prepare Polaris for connections

Before you send data from Decodable into Imply Polaris, do the following in your Polaris account to make sure that you are able to create a connection to it.

  • Create a table in Polaris. See Create a table by API for instructions on how to create a table.

  • Create a Polaris push_streaming connection and a streaming job to your Polaris table. The input schema specified in the streaming job and the schema of the Decodable stream that is sending data to Polaris must match. Otherwise, your data may not appear in the Polaris table. See Create a table by API for instructions on how to create Polaris connections and jobs.

    • Make sure that an event timestamp field exists in the Decodable stream schema. You must map the timestamp field in Decodable to the timestamp field in the Polaris streaming job.

Steps: Create a connection with the Imply Polaris connector

Follow these steps to send data from Decodable into a Polaris table. These steps assume that you are using Decodable Web. However, if you want to use the Decodable REST API to create the connection, see Example: Use the REST API to send data from Decodable to Imply Polaris.

  1. From the Connections page, select the Imply Polaris Connector and complete the following fields.

    Field Description

    Connection Name

    The name of the push_streaming connection associated with the Polaris table you want to send data to. If you want to send data to a Polaris v1 table, then this is the same as the Table ID.

    Organization Name

    The name of the organization containing the Polaris table you want to send data to.

    Client ID

    The OAuth client ID obtained from Polaris.

    Client Secret

    The client secret associated with the OAuth client ID.

    Cloud Provider

    The cloud service provider for your Polaris project. Currently, only aws is supported.

    Region

    The region that the Polaris project is in. The following regions are supported:

    • us-east-1

    • eu-central-1

  2. Select which stream contains the records that you’d like to send to Polaris. Then, select Next.

  3. Give the newly created connection a Name and Description and select Save.

Start this connection to begin sending data from Decodable to Polaris.

Example: Use the REST API to send data from Decodable to Imply Polaris

Let’s walk through an example of the four steps above. We will use the Polaris REST API to illustrate connection and job creation in Polaris. We will also use the Decodable REST API to illustrate creating the sink connection to Imply Polaris. (NOTE: this workflow can also be done via the UI/CLI of both services as well).

In this example, assume that we’ve already created a Polaris table called http_requests with the following schema:

       |        |               |               |
__time | method | original_path | response_code | bytes_rcvd
       |        |               |               |

In addition, we’ve also already created a Decodable stream called http_requests_stream with the following schema:

          |        |               |               |
timestamp | method | original_path | response_code | bytes_rcvd
          |        |               |               |
Notice in this example that the timestamp field is called _time in the Polaris table and timestamp in the Decodable stream. We will create a mapping between these two fields in the streaming job as part of step 1.

Create a push_streaming connection and a streaming job using the Polaris REST API.

curl --location --request POST 'https://ORGANIZATION_NAME.api.imply.io/v2/connections' \
--header 'Authorization: Bearer $IMPLY_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
  "type": "push_streaming",
  "name": "demo_connection"
}'
curl --location --request POST 'https://ORGANIZATION_NAME.api.imply.io/v2/jobs' \
--header "Authorization: Bearer $IMPLY_TOKEN" \
--header 'Content-Type: application/json' \
--data-raw '{
    "type": "streaming",
    "target": {
        "type": "table",
        "tableName": "http_requests"
    },
    "source": {
        "type": "connection",
        "connectionName": "demo_connection",
        "inputSchema": [
            {
                "name": "timestamp",
                "dataType": "string"
            },
            {
                "name": "method",
                "dataType": "string"
            },
            {
                "name": "original_path",
                "dataType": "string"
            },
            {
                "name": "response_code",
                "dataType": "long"
            },
            {
                "name": "bytes_rcvd",
                "dataType": "long"
            }
        ],
        "formatSettings": {
            "format": "nd-json"
        }
    },
    "mappings": [
        {
            "columnName": "__time",
            "expression": "TIME_PARSE(\"timestamp\")"
        },
        {
            "columnName": "method",
            "expression": "\"method\""
        },
        {
            "columnName": "original_path",
            "expression": "\"original_path\""
        },
        {
            "columnName": "response_code",
            "expression": "\"response_code\""
        },
        {
            "columnName": "bytes_rcvd",
            "expression": "\"bytes_rcvd\""
        }
    ]
}'

Create a connection to Imply Polaris using the Decodable REST API.

curl --request POST \
     --url https://<your_decodable_organization>.api.decodable.co/v1alpha2/connections \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer $DECODABLE_AUTH_TOKEN' \
     --header 'Content-Type: application/json' \
     --data '
{
     "connector": "polaris",
     "properties": {
          "polaris.client-id": "<your_client_ID>",
          "polaris.connection-name": "<your_connection_name>",
          "polaris.org-name": "<your_org_name>",
          "polaris.client-secret": "<your_client_secret>",
          "polaris.cloud": "<cloud_provider">,
          "polaris.region": "<region>"
     },
     "name": "imply_polaris_demo",
     "description": "Send data to Imply Polaris from Decodable",
     "type": "sink",
     "stream_id": "<http_requests_stream ID>"
}
'

Start the imply_polaris_demo connection from the REST API/UI/CLI and sit back, pet your cat, take a walk, or start working with your data in Polaris!

Apache Kafka, Kafka®, Apache®, Druid®, and associated open source project names are either registered trademarks or trademarks of The Apache Software Foundation.