Google Cloud Pub/Sub

Use the Google Cloud Pub/Sub (GCP Pub/Sub) to get data from Google Cloud into Decodable. If you are looking for instructions on how to send data from Decodable into Google Cloud, see Google Cloud Platform Pub/Sub in the Connect to a data destination chapter.

Overview

Connector name

gcp-pubsub

Type

source, sink

Delivery guarantee

at least once

Prerequisites

You must have the following.

  • A GCP service account key that can be used for authentication. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM.

  • A GCP subscription with the Pull delivery type.

  • The following GCP roles assigned to the service account principal:

    GCP Resource Role 1 Role 2

    Subscription

    Pub/Sub Viewer

    Pub/Sub Subscriber

Steps

Follow these steps to send data from Google Cloud Pub/Sub into Decodable. These steps assume that you are using Decodable Web. However, if you want to use the Decodable CLI to create the connection, you can refer to the Property Name column for information about what the underlying property names are.

  1. From the Connections page, select the GCP Pub/Sub connector and complete the following fields.

    UI Field Property Name in the Decodable CLI Description

    Connection Type

    N/A

    Set to Source to create a connection that gets data from GCP Pub/Sub into Decodable.

    Project

    project

    The project name that the Pub/Sub resource is located in.

    Subscription

    subscription

    The name of the subscription to read data from.

    Service Key JSON Secret

    service-key-json

    The secret associated with your service account. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

    Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Manage Roles, Groups, and Permissions for more information.

    Value Format

    format

    The format of the data in the stream. Must be json.

    N/A

    subscription.max-messages-per-pull

    The maximum number of pulled messages per pull request. This property is only configurable through the Decodable CLI.

    Defaults to 100.

    N/A

    subscription.per-request-timeout-sec

    Optional. The amount of time, in seconds, to wait before timing out. This property is only configurable through the Decodable CLI.

    Defaults to 15 seconds.

    N/A

    subscription.retries

    Optional. The maximum number of subscription retry attempts. This property is only configurable through the Decodable CLI.

    Defaults to 3.

    Messages are acknowledged to Pub/Sub approximately every 10 seconds. Best practices are to set the subscription’s acknowledgement deadline to be much larger, or else the messages might be processed multiple times.
  2. Select the stream that you’d like to connect to this connector. Then, select Next.

  3. Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Structured Schema Definition if you want to paste the schema in the form of an Avro or JSON array.

    1. The stream’s schema must match the schema of the data that you plan on sending through this connection.

    2. For more information about creating a stream or defining the stream schema, see Create and manage streams.

  4. Select Next when you are finished providing defining the connection’s schema.

  5. Give the newly created connection a Name and Description and select Save.