Apache Pulsar

Apache® Pulsar™ is a cloud-native, multi-tenant, high-performance solution for server-to-server messaging and queuing built on the publisher-subscribe (pub-sub) pattern. Pulsar combines the best features of a traditional messaging system like RabbitMQ with those of a pub-sub system like Apache Kafka, scaling up or down dynamically without downtime. It's used by thousands of companies for high-performance data pipelines, microservices, instant messaging, data integrations, and more.

  • Cloud-native
  • Serverless functions
  • Horizontally scalable
  • Low latency with durability
  • Geo-replication
  • Multi-tenancy

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kafka connectors can be used in either role, performing as either a source or a sink. The configuration is almost identical, except that source connectors allow for defining a policy for handling parse errors of the incoming data stream.

Configure As A Source

To create and configure a connector for Kafka, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

  1. Select a connector type, either source or sink, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system.

  2. Specify the broker service URL that will be used to connect to your Pulsar broker. For example, pulsar+ssl://broker.example.com:6651.

  3. Specify the admin web service URL that will be used to connect to your Pulsar admin endpoint. For example, <https://broker.example.com>.

  4. Provide the name of your Pulsar topic.

  5. Optionally provide the JWT token if one is needed for authentication.

  6. The value format will default to JSON, since that is the only option for Kinesis connectors.

For more detailed information about Apache Pulsar, see the Pulsar Getting Started guide and related documentation.

Configure As A Sink

To create a Pulsar connector for use as a sink, all the configuration steps above apply.

Reference

Connector namepulsar
Typesource, sink
Delivery guaranteeat least once
CompatibilityPulsar 2.9.x or above

The Pulsar connector streams data into or out of an Apache Pulsar® topic.

Properties

The following properties are supported by the Kinesis connector.

PropertyDispositionDescription
topicrequiredFully qualified pulsar topic name
Example: persistent://decodable/default/test
service-urlrequiredThe broker service URL.
Example: pulsar+ssl://broker.example.com:6651
admin-urlrequiredThe admin web service URL.
Example: <https://broker.example.com>
formatrequiredMust be json
tokenoptionalJWT token for authentication.
Example: eyJhbGciOiJIUzI1NiJ9...

JSON Format Properties

When using format=json, the following properties are optionally allowed:

PropertyDescription
json.timestamp-format.standardSpecify the timestamp format for TIMESTAMP and TIMESTAMP_LTZ types.
  • SQL will use a "yyyy-MM-dd HH:mm:ss.SSS" format, e.g "2020-12-30 12:13:14.123"
  • ISO-8601 will parse input TIMESTAMP in "yyyy-MM-ddTHH:mm:ss.SSS" format, e.g "2020-12-30T12:13:14.123"
json.encode.decimal-as-plain-numberMust be true or false, defaults to false.
When true, always encode numbers without scientific notation.
For example, a number encoded 2.7E-8 by default would be encoded 0.000000027.

Apache® Pulsar™, Apache, Pulsar, and associated open source project names are either registered trademarks or trademarks of The Apache Software Foundation.

Did this page help you?