DataStax Astra Streaming

Astra Streaming is a multi-cloud streaming-as-a-service product built on Apache Pulsar by DataStax. Pulsar is a cloud-native, multi-tenant, high-performance solution for server-to-server messaging and queuing built on the publisher-subscribe (pub-sub) pattern. Pulsar combines the best features of a traditional messaging system like RabbitMQ with those of a pub-sub system like Apache Kafka, scaling up or down dynamically without downtime. It's used by thousands of companies for high-performance data pipelines, microservices, instant messaging, data integrations, and more.

DataStax delivers Pulsar through its Astra Streaming product, which provides a fully-managed SaaS with boundless scale, massive throughput, and low latency.

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system (Astra stream, in this case) and write to a Decodable stream, while sink connections read from a stream and write to an external system (Astra stream). Astra connectors can be used in either role, performing as either a source or a sink. The configuration is identical for both roles.

Configure As A Source

To create and configure a connector for Astra Streaming, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

  1. Select a connector type, either source or sink, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system.

  2. Specify the broker service URL that will be used to connect to your Astra broker. For example, pulsar+ssl://broker.example.com:6651.

  3. Specify the admin web service URL that will be used to connect to your Astra admin endpoint. For example, <https://broker.example.com>.

  4. Provide the fully-qualified name of your Astra topic. For example, persistent://stream/namespace/topic-name.

  5. Provide the JWT token needed for authentication to your Astra stream.

  6. The value format will default to JSON, since that is the only option for Astra connectors.

For more detailed information about DataStax Astra Streaming, see the product website for related documentation.

Configure As A Sink

To create an Astra connector for use as a sink, all the configuration steps above apply.

Reference

Connector nameastra-streaming
Typesource, sink
Delivery guaranteeat least once
CompatibilityPulsar 2.9.x or above

Properties

The following properties are supported by the Kinesis connector.

PropertyDispositionDescription
topicrequiredFully qualified Astra streaming topic name
Example: persistent://decodable/default/test
service-urlrequiredThe broker service URL.
Example: pulsar+ssl://broker.example.com:6651
admin-urlrequiredThe admin web service URL.
Example: <https://broker.example.com>
formatrequiredMust be json
tokenrequiredJWT token for authentication.
Example: eyJhbGciOiJIUzI1NiJ9...