Elasticsearch sink connector

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. You can use Decodable to ingest records (or documents) to your Elasticsearch cluster and enable your data to be searchable in seconds.

Features

Delivery guarantee

At least once

Supported stream types

append, change

Supported versions

  • Elasticsearch 7.x

  • Elasticsearch 6.x

Steps

If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is elasticsearch.

  1. From the Connections page, select the Elasticsearch connector and complete the following fields.

    UI Field

    Property Name

    Description

    Version

    version

    The version of Elasticsearch to which you are connecting.

    INFO: Make sure to specify the correct version of Elasticsearch you are using when configuring your sink connection to avoid unexpected compatibility errors.

    Hosts

    hosts

    A semicolon-delimited list of Elasticsearch hosts to connect to. For example:

    • 'https://my-es-cluster.com:443'

      or

    • 'http://es-node-1:9092;http://es-node-2:9093'

    While the Elasticsearch sink does support endpoints using TLS/SSL encryption, it does not currently support client certificate authentication or specifying additional TLS/SSL options.

    Index

    index

    The name of the Elasticsearch index to send data to. If the index doesn’t exist, Decodable will create it for you.

    Decodable supports static and dynamic index creation. See Types of Index.

    Username

    username

    The username to connect to your Elasticsearch cluster.

    Password

    password

    The authentication token associated with your Elasticsearch password. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

    For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Roles, groups, and permissions for more information.

    Document type

    document-type

    (Elasticsearch v6 only)

    The Elasticsearch document mapping type.

    WARNING: Modifying the document-type of a connection that has already ingested records to an index will result in runtime errors. If you change the document-type, make sure to change or delete the index as well.

  2. Select which stream contains the records that you’d like to send to Elasticsearch. Then, select Next.

  3. Give the newly created connection a Name and Description and select Save.

You can now start the connection to send data from Decodable into your Elasticsearch cluster.

Types of Index

  • Static index: a single target index that all records will be continuously written into represented by a plain string index name. For example, users.

  • Dynamic index: includes {field_name} in the index name to reference a field value and dynamically generate a target index for each record.

    The dynamic index name can also be of the form {field_name|date_format_string}, where date_format_string is a format pattern compatible with Java’s DateTimeFormatter. For example:

    • users-at-{ts|yyyy-MM-dd}, where ts is a timestamp field in the connector schema and yyyy-MM-dd is an optional date format pattern.

    • 'index-for-user-{user}'

When using a dynamic index generated by the current system time with a change stream, there is no guarantee that the records with the same primary key can generate the same index name. So, the dynamic index based on the system time can only support append streams.