Elasticsearch

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. You can use Decodable to ingest records (or documents) to your Elasticsearch cluster and enable your data to be searchable in seconds.

Overview

Connector nameelasticsearch
Typesink
Delivery guaranteeat least once
Supported stream typesappend, change

Supported Versions

Decodable currently supports the following versions of Elasticsearch:

  • v7.x
  • v6.x

Please ensure you specify the correct version of Elasticsearch you are using when configuring your sink connection to avoid unexpected compatibility errors.

Configure As A Sink

Elasticsearch is a sink connection, meaning that data can only be written from Decodable to Elasticsearch. Sending a Decodable stream to Elasticsearch can be accomplished in 3 steps:

  1. (Optional) Create an index in your Elasticsearch cluster. Decodable can also do this on your behalf given an index name. Decodable supports static and dynamic index creation:
    • static index : a single target index that all records will be continuously written into represented by a plain string index name.
      • ex: 'users'
    • dynamic index: includes {field_name} in the index name to reference a field value and dynamically generate a target index for each record. The dynamic index name can also be of the form {field_name|date_format_string}, where date_format_string is a format pattern compatible with Java’s DateTimeFormatter.
      • ex: 'users-at-{ts|yyyy-MM-dd}', where ts is a timestamp field in the connector schema and yyyy-MM-dd is an optional date format pattern.
      • ex: 'index-for-user-{user}'
  2. Create a Decodable stream to ingest to the Elasticsearch index from step 1.
    Note: The Elasticsearch sink supports both append and change streams.
  3. Create the connection by specifying:
    1. hosts: A single hostname or a semicolon-delimited list of hostnames that identify your Elasticsearch cluster.
      • ex: 'https://my-es-cluster.com:443'
      • ex: 'http://es-node-1:9092;http://es-node-2:9093'
    2. index: Your index name from step 1.
    3. (ES v6 only)document-type: The Elasticsearch document mapping type.
      Note: this has been removed in Elasticsearch 7.x onwards.
    4. (Optional) version: Your cluster's Elasticsearch version.
    5. (Optional) username: The username with which to connect to your Elasticsearch cluster.
    6. (Optional) password: The password for authentication as the username.

Start the connection and observe data flow into your Elasticsearch cluster.

Notes

  • While the Elasticsearch sink does support endpoints using TLS/SSL encryption, it does not currently support client certificate authentication or specifying additional TLS/SSL options.
  • When using a dynamic index generated by the current system time with a change stream, there is no guarantee that the records with the same primary key can generate the same index name. So, the dynamic index based on the system time can only support append streams.
  • (ES v6) Modifying the document-type of a connection that has already ingested records to an index will result in runtime errors. If you change the document-type, be sure to change or delete the index as well.