Apache Pulsar
Apache® Pulsar™ is a cloud-native, multi-tenant, high-performance solution for server-to-server messaging and queuing built on the publisher-subscribe (pub-sub) pattern. Pulsar combines the best features of a traditional messaging system like RabbitMQ with those of a pub-sub system like Apache Kafka, scaling up or down dynamically without downtime. It's used by thousands of companies for high-performance data pipelines, microservices, instant messaging, data integrations, and more.
- Cloud-native
- Serverless functions
- Horizontally scalable
- Low latency with durability
- Geo-replication
- Multi-tenancy
Getting Started
Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Pulsar connectors can be used in either role, performing as either a source
or a sink
. The configuration is identical for both roles.
Configure As A Source
To create and configure a connector for Pulsar, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection
, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.
-
Select a connector type, either
source
orsink
, which will determine whether data is being streamed into Decodable for processing by one or more pipelines, or streamed out to an external system. -
Specify the broker service URL that will be used to connect to your Pulsar broker. For example,
pulsar+ssl://broker.example.com:6651
. -
Specify the admin web service URL that will be used to connect to your Pulsar admin endpoint. For example,
<https://broker.example.com
>. -
Provide the name of your Pulsar topic.
-
Optionally provide the JWT token if one is needed for authentication.
-
The value format will default to
JSON
, since that is the only option for Pulsar connectors.
For more detailed information about Apache Pulsar, see the Pulsar Getting Started guide and related documentation.
Configure As A Sink
To create a Pulsar connector for use as a sink
, all the configuration steps above apply.
Reference
Connector name | pulsar |
Type | source , sink |
Delivery guarantee | at least once |
Compatibility | Pulsar 2.9.x or above |
The Pulsar connector streams data into or out of an Apache Pulsar® topic.
Properties
The following properties are supported by the Kinesis connector.
Property | Disposition | Description |
---|---|---|
topic | required | Fully qualified pulsar topic name Example: persistent://decodable/default/test |
service-url | required | The broker service URL. Example: pulsar+ssl://broker.example.com:6651 |
admin-url | required | The admin web service URL. Example: <https://broker.example.com > |
format | required | Must be json |
token | optional | JWT token for authentication. Example: eyJhbGciOiJIUzI1NiJ9... |
JSON Format Properties
When using format
=json
, the following properties are optionally allowed:
Property | Description |
---|---|
json.timestamp-format.standard | Specify the timestamp format for TIMESTAMP and TIMESTAMP_LTZ types.
|
json.encode.decimal-as-plain-number | Must be true or false , defaults to false .When true , always encode numbers without scientific notation.For example, a number encoded 2.7E-8 by default would be encoded 0.000000027 . |
Updated 4 months ago