OpenSearch sink connector

Use the OpenSearch connector to send data from Decodable to OpenSearch, an open source search and analytics suite.

Features

Delivery Guarantee

at least once

Supported versions

  • OpenSearch 1.x

  • OpenSearch 2.x

Steps

If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is opensearch.
  1. From the Connections page, select the OpenSearch connector and complete the following fields.

    UI Field Property Name Description

    Hosts

    hosts

    A semicolon-delimited list of OpenSearch hosts to connect to. For example: http://host_name:9092;http://host_name:9093.

    Index

    index

    The name of the OpenSearch index to send data to. If the index doesn’t exist, Decodable will create it for you. Decodable supports two different methods for sending data to an OpenSearch index:

    Send data to a specific index: Send data to a single target index by specifying a plain string index name. For example: users.

    Dynamically route data to different indexes: Send data to specific indexes based on a field value present in the records themselves. To use this option, include the field name, enclosed in brackets, that you want to route data based on. You can also include a date format patterns with the field name. See DateTimeFormatter for supported format patterns. For example:

    index-for-user-\{user\} where user is a field present in the stream schema. users-at-{ts|yyyy-MM-dd} where ts is a timestamp field present in the stream schema and yyyy-MM-dd is an optional date format pattern.

    Username

    username

    The username to connect to your OpenSearch cluster.

    Password

    password

    The authentication token associated with your OpenSearch password. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

    Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Roles, groups, and permissions for more information.

     — 

    sink.bulk-flush.max-actions

    Advanced. Maximum number of buffered records per request.

    Defaults to 1000.

    NOTE: This configuration is currently available only through the CLI.

     — 

    sink.bulk-flush.max-size

    Advanced. Maximum size of buffered records per request.

    Defaults to 2mb.

    NOTE: This configuration is currently available only through the CLI.

     — 

    sink.bulk-flush.interval

    Advanced. The interval to flush buffered records.

    Defaults to 1s.

    NOTE: This configuration is currently available only through the CLI.

  2. Select which stream contains the records that you’d like to send to OpenSearch. Then, select Next.

  3. Give the newly created connection a Name and Description and select Save.

You can now start the connection to send data from Decodable into your OpenSearch cluster.

Connector starting state and offsets

A new sink connection will start reading from the Latest point in the source Decodable stream. This means that only data that’s written to the stream when the connection has started will be sent to the external system. You can override this when you start the connection to Earliest if you want to send all the existing data on the source stream to the target system, along with all new data that arrives on the stream.

When you restart a sink connection it will continue to read data from the point it most recently stored in the checkpoint before the connection stopped. You can also opt to discard the connection’s state and restart it afresh from Earliest or Latest as described above.

Learn more about starting state here.