Datastax Astra Streaming source connector

Use the DataStax Astra Streaming Connector to get data from an Astra stream. If you want to send data to Datastax Astra, see DataStax Astra Streaming.

Features

Delivery guarantee

At least once

Compatibility

Pulsar 2.9.x or above

Steps

If you want to use the Decodable CLI or API to create the connection, you can refer to the Property Name column for information about what the underlying property names are. The connector name is astra-streaming.

  1. From the Connections page, select the DataStax Astra Streaming Connector and complete the following fields.

    UI Field Property Name Description

    Connection Type

    N/A

    Select source to send data from an Astra topic into Decodable.

    Broker Service URL

    service-url

    The URL to connect to your Astra Streaming broker.

    For example, pulsar+ssl://broker.example.com:6651.

    Web Service URL

    admin-url

    The URL to connect to your Astra Streaming admin endpoint.

    For example, <https://broker.example.com>.

    Topic

    topic

    The fully-qualified name of the Astra topic.

    For example, persistent://stream/namespace/topic-name.

    Authentication Token

    token

    The authentication token associated with your Astra stream. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

    Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Roles, groups, and permissions for more information.

    Partition Key Columns

    key.fields

    A list of fields, delimited by semicolons, that comprise the partition key.

    For example: field1;field2.

    Partition Key Format

    key.format

    Optional. The format used to serialize and deserialize the partition key. Must be one of the following:

    • JSON

    • Avro

    • Raw

    Value Format

    format

    The format for data in the Astra topic. Must be one of the following:

    • JSON

    • AVRO

    • Raw

    • Debezium (JSON)

    If you want to send CDC data through this connector, then you must select Debezium (JSON).

    Partition Key Inclusion Policy

    value.fields-include

    Specify whether the partition key columns should be present in the payload values sent to Decodable.

    For an example of how the key.fields, key.format, and value.fields-include arguments work together, see the examples in the Key and Value Formats section in the Apache Flink documentation.

    Scan Startup Mode

    scan.startup.mode

    Specifies where in the topic to start reading data when the connection is first started or after discarding state.

    • latest: Start reading data from the latest available point in the stream.

    • earliest: Start reading data from the earliest available point in the stream.

  2. Select the stream that you’d like to connect to this connector. Then, select Next.

  3. Define the connection’s schema. Decodable can auto-populate the connection’s schema using Pulsar’s schema registry. In most cases, you’ll want to select Schema Registry to automatically populate the connection’s schema. However, if you would like to manually enter the schema, select New Schema or Import Schema.

    1. If you want to auto-populate the connection schema using Pulsar’s schema registry, you’ll also need to provide the name of your Pulsar tenant, the namespace where your topic is, and the version of the schema that you’d like to use.

  4. Select Next when you are finished providing defining the connection’s schema.

  5. Give the newly created connection a Name and Description and select Save.