Google Cloud Pub/Sub sink connector

Features

Delivery guarantee

At least once

Prerequisites

You must have the following.

  • A GCP service account key that can be used for authentication. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM.

  • The following GCP roles assigned to the service account principal:

    GCP Resource Role 1 Role 2

    Subscription

    Pub/Sub Viewer

    Pub/Sub Subscriber

Steps

To create and configure a connector for Pub/Sub, sign in to Decodable Web, navigate to the Connections tab, click on New Connection, and follow the steps below. For examples of using the command line tools or scripting, see the How To guides.

  1. Select a connector type source, where data will be read from Google Pub/Sub to a Decodable stream.

  2. Provide the project name containing the Pub/Sub resource.

  3. Provide the subscription name of your Pub/Sub topic. Note that the subscription’s delivery type must be 'Pull'.

  4. Provide the JSON service key secret for the service account to access Pub/Sub.

  5. Specify the value format to read message payloads from Pub/Sub.

Messages are acknowledged to Pub/Sub approximately every 10 seconds. Therefore best practices are to set the subscription’s acknowledgement deadline to be much larger, or else the messages might be processed multiple times.

Configure as a sink

  1. Select a connector typesink, where the contents of a stream will be written to a Pub/Sub topic.

  2. Provide the project name containing the Pub/Sub resource.

  3. Provide the topic name of your Pub/Sub topic.

  4. Provide the JSON service key secret for the service account to access Pub/Sub.

  5. Specify the value format to write message payloads to Pub/Sub.

Properties

If you want to use the Decodable CLI or API to create the connection, you can refer to the Property column for information about what the underlying property names are. The connector name is gcp-subpub.

The following properties are supported by the Pub/Sub connector.

Property Disposition Description

project

required

The project name the Pub/Sub resource is located in

topic

required (sink)

The topic name to write events to

subscription

required (source)

The subscription name to read events from

subscription.max-messages-per-pull

optional

Max messages per pull request. Default: 100

subscription.per-request-timeout-sec

optional

Subscription request timeout (s). Default: 15

subscription.retries

optional

Subscription retry limit. Default: 3