GCP Pub/Sub
Google Cloud Pub/Sub is an asynchronous, scalable message queue service that decouples services producing messages from services receiving them. Publishers write write messages to topics, and subscribers receive these messages. It is frequently used to ingest data into Google BigQuery, data lakes, or other operational databases.
Getting Started
Connections come in two flavors: source and sink. Source connections read from an external system and write to a Decodable stream, while sink connections read from a stream and write to an external system. Pub/Sub connectors can be used in either role, performing as either a source
or a sink
.
Configure As A Source
To create and configure a connector for Pub/Sub, sign in to the Decodable Web Console, navigate to the Connections tab, click on New Connection
, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.
-
Select a connector type
source
, where data will be streamed into Decodable for processing by one or more pipelines -
Provide the project name containing the Pub/Sub resource
-
Provide the subscription name of your Pub/Sub topic. Note that the subscription's delivery type must be 'Pull'.
-
Provide the JSON service key secret for the service account to access Pub/Sub
-
Specify the value format to read message payloads from Pub/Sub
Subscription Deadlines
Messages are acknowledged to Pub/Sub approximately every 10 seconds. Therefore it is recommended to set the subscription's acknowledgement deadline to be much larger, or else the messages may be processed multiple times.
Configure As A Sink
-
Select a connector type
sink
, where the contents of a stream will be written to a Pub/Sub topic -
Provide the project name containing the Pub/Sub resource
-
Provide the topic name of your Pub/Sub topic
-
Provide the JSON service key secret for the service account to access Pub/Sub
-
Specify the value format to write message payloads to Pub/Sub
Reference
Connector name | gcp-pubsub |
Type | source , sink |
Delivery guarantee | at least once |
The Pub/Sub connector uses a service account key to authenticate against Google Cloud. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM. It is a JSON file with the first field being "type": "service_account"
The service account principal requires the following GCP Roles to be assigned:
Connection Type | GCP Resource | Role 1 | Role 2 |
---|---|---|---|
Source | Subscription | Pub/Sub Viewer | Pub/Sub Subscriber |
Sink | Topic | Pub/Sub Viewer | Pub/Sub Publisher |
Properties
The following properties are supported by the Pub/Sub connector.
Property | Disposition | Description |
---|---|---|
project | required | The project name the Pub/Sub resource is located in |
topic | required (sink) | The topic name to write events to |
subscription | required (source) | The subscription name to read events from |
subscription.max-messages-per-pull | optional | Max messages per pull request. Default: 100 |
subscription.per-request-timeout-sec | optional | Subscription request timeout (s). Default: 15 |
subscription.retries | optional | Subscription retry limit. Default: 3 |
Updated 5 months ago