Apache® Pulsar™ is a cloud-native, multi-tenant, high-performance solution for server-to-server messaging and queuing built on the publisher-subscribe (pub-sub) pattern. Pulsar combines the best features of a traditional messaging system like RabbitMQ with those of a pub-sub system like Apache Kafka, scaling up or down dynamically without downtime. It's used by thousands of companies for high-performance data pipelines, microservices, instant messaging, data integrations, and more.

Cloud-native
Serverless functions
Horizontally scalable
Low latency with durability
Geo-replication
Multi-tenancy

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Kafka connectors can be used in either role, performing as either a source or a sink. The configuration is almost identical, except that source connectors allow for defining a policy for handling parse errors of the incoming data stream.

Configure As A Source

To create and configure a connector for Kafka, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

Select a connector type, either source or sink, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system.
Specify the broker service URL that will be used to connect to your Pulsar broker. For example, pulsar+ssl://broker.example.com:6651.
Specify the admin web service URL that will be used to connect to your Pulsar admin endpoint. For example, <https://broker.example.com>.
Provide the name of your Pulsar topic.
Optionally provide the JWT token if one is needed for authentication.
The value format will default to JSON, since that is the only option for Kinesis connectors.

For more detailed information about Apache Pulsar, see the Pulsar Getting Started guide and related documentation.

Configure As A Sink

To create a Pulsar connector for use as a sink, all the configuration steps above apply.

Reference


Connector name	`pulsar`
Type	`source`, `sink`
Delivery guarantee	at least once
Compatibility	Pulsar 2.9.x or above

The Pulsar connector streams data into or out of an Apache Pulsar® topic.

Properties

The following properties are supported by the Kinesis connector.

Property	Disposition	Description
`topic`	required	Fully qualified pulsar topic name Example: `persistent://decodable/default/test`
`service-url`	required	The broker service URL. Example: `pulsar+ssl://broker.example.com:6651`
`admin-url`	required	The admin web service URL. Example: `<https://broker.example.com`>
`format`	required	Must be `json`
`token`	optional	JWT token for authentication. Example: `eyJhbGciOiJIUzI1NiJ9...`

JSON Format Properties

When using format=json, the following properties are optionally allowed:

Property	Description
`json.timestamp-format.standard`	Specify the timestamp format for `TIMESTAMP` and `TIMESTAMP_LTZ` types. `SQL` will use a "yyyy-MM-dd HH:mm:ss.SSS" format, e.g "2020-12-30 12:13:14.123" `ISO-8601` will parse input `TIMESTAMP` in "yyyy-MM-ddTHH:mm:ss.SSS" format, e.g "2020-12-30T12:13:14.123"
`json.encode.decimal-as-plain-number`	Must be `true` or `false`, defaults to `false`. When `true`, always encode numbers without scientific notation. For example, a number encoded `2.7E-8` by default would be encoded `0.000000027`.

Apache® Pulsar™, Apache, Pulsar, and associated open source project names are either registered trademarks or trademarks of The Apache Software Foundation.