Google Cloud Pub/Sub is an asynchronous, scalable message queue service that decouples services producing messages from services receiving them. Publishers write write messages to topics, and subscribers receive these messages. It is frequently used to ingest data into Google BigQuery, data lakes, or other operational databases.

Getting Started

Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Pub/Sub connectors can be used in either role, performing as either a source or a sink.

Configure As A Source

To create and configure a connector for Pub/Sub, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

1. Select a connector type source, where data will be streamed into Decodable for processing by one or more pipelines

2. Provide the project name containing the Pub/Sub resource

3. Provide the subscription name of your Pub/Sub topic. Note that the subscription's delivery type must be 'Pull'.

4. Provide the JSON service key secret for the service account to access Pub/Sub

5. Specify the value format to read message payloads from Pub/Sub

📘

Subscription Deadlines

Messages are acknowledged to Pub/Sub approximately every 10 seconds. Therefore it is recommended to set the subscription's acknowledgement deadline to be much larger, or else the messages may be processed multiple times.

Configure As A Sink

1. Select a connector typesink, where the contents of a stream will be written to a Pub/Sub topic

2. Provide the project name containing the Pub/Sub resource

3. Provide the topic name of your Pub/Sub topic

4. Provide the JSON service key secret for the service account to access Pub/Sub

5. Specify the value format to write message payloads to Pub/Sub

Reference

The Pub/Sub connector uses a service account key to authenticate against Google Cloud. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM. It is a JSON file with the first field being "type": "service_account"

The service account principal requires the following GCP Roles to be assigned:

Properties

The following properties are supported by the Pub/Sub connector.