Google Cloud Pub/Sub

Use the Google Cloud Pub/Sub (GCP Pub/Sub) to get data from Google Cloud into Decodable. If you are looking for instructions on how to send data from Decodable into Google Cloud, see Google Cloud Platform Pub/Sub in the Connect to a data destination chapter.

Overview

Connector namegcp-pubsub
Typesource, sink
Delivery guaranteeat least once

Create a connection to Google Cloud Pub/Sub

Prerequisites

You must have the following.

  • A GCP service account key that can be used for authentication. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM.

  • A GCP subscription with the Pull delivery type.

  • The following GCP roles assigned to the service account principal:

    GCP ResourceRole 1Role 2
    SubscriptionPub/Sub ViewerPub/Sub Subscriber

Steps

Follow these steps to send data from Google Cloud Pub/Sub into Decodable. These steps assume that you are using Decodable Web. However, if you want to use the Decodable CLI to create the connection, you can refer to the Property Name column for information about what the underlying property names are.

  1. From the Connections page, select the GCP Pub/Sub connector and complete the following fields.
UI FieldProperty Name in the Decodable CLIDescription
Connection TypeN/ASet to Source to create a connection that gets data from GCP Pub/Sub into Decodable.
ProjectprojectThe project name that the Pub/Sub resource is located in.
SubscriptionsubscriptionThe name of the subscription to read data from.
Service Key JSON Secretservice-key-jsonThe secret associated with your service account. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.

Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Manage Roles, Groups, and Permissions for more information.
Value FormatformatThe format of the data in the stream. Must be json.
N/Asubscription.max-messages-per-pullThe maximum number of pulled messages per pull request. This property is only configurable through the Decodable CLI.

Defaults to 100.
N/Asubscription.per-request-timeout-secOptional. The amount of time, in seconds, to wait before timing out. This property is only configurable through the Decodable CLI.

Defaults to 15 seconds.
N/Asubscription.retriesOptional. The maximum number of subscription retry attempts. This property is only configurable through the Decodable CLI.

Defaults to 3.

📘

Subscription Deadlines

Messages are acknowledged to Pub/Sub approximately every 10 seconds. Best practices are to set the subscription's acknowledgement deadline to be much larger, or else the messages may be processed multiple times.

  1. Select the stream that you’d like to connect to this connector. Then, select Next.

  2. Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Structured Schema Definition if you want to paste the schema in the form of an Avro or JSON array.

    1. The stream's schema must match the schema of the data that you plan on sending through this connection.
    2. For more information about creating a stream or defining the stream schema, see Create a Stream.
  3. Select Next when you are finished providing defining the connection’s schema.

  4. Give the newly created connection a Name and Description and select Save.