Google Cloud Pub/Sub
Use the Google Cloud Pub/Sub (GCP Pub/Sub) to get data from Google Cloud into Decodable. If you are looking for instructions on how to send data from Decodable into Google Cloud, see Google Cloud Platform Pub/Sub in the Connect to a data destination chapter.
Overview
Connector name | gcp-pubsub |
Type | source , sink |
Delivery guarantee | at least once |
Create a connection to Google Cloud Pub/Sub
Prerequisites
You must have the following.
-
A GCP service account key that can be used for authentication. You can create and download the key by browsing to the Keys tab under the Service Account details in GCP IAM.
-
A GCP subscription with the
Pull
delivery type. -
The following GCP roles assigned to the service account principal:
GCP Resource Role 1 Role 2 Subscription Pub/Sub Viewer Pub/Sub Subscriber
Steps
Follow these steps to send data from Google Cloud Pub/Sub into Decodable. These steps assume that you are using Decodable Web. However, if you want to use the Decodable CLI to create the connection, you can refer to the Property Name column for information about what the underlying property names are.
- From the Connections page, select the GCP Pub/Sub connector and complete the following fields.
UI Field | Property Name in the Decodable CLI | Description |
---|---|---|
Connection Type | N/A | Set to Source to create a connection that gets data from GCP Pub/Sub into Decodable. |
Project | project | The project name that the Pub/Sub resource is located in. |
Subscription | subscription | The name of the subscription to read data from. |
Service Key JSON Secret | service-key-json | The secret associated with your service account. If you are using the Decodable CLI, this is the ID of a secret resource in your account. Run decodable secret list to view available secrets or decodable secret --help for help with creating a new secret.Note: For security purposes, Decodable will never display secret values in plaintext. You can manage which users have permissions to create, delete, or modify secrets in the Access Control management view. See Manage Roles, Groups, and Permissions for more information. |
Value Format | format | The format of the data in the stream. Must be json . |
N/A | subscription.max-messages-per-pull | The maximum number of pulled messages per pull request. This property is only configurable through the Decodable CLI. Defaults to 100. |
N/A | subscription.per-request-timeout-sec | Optional. The amount of time, in seconds, to wait before timing out. This property is only configurable through the Decodable CLI. Defaults to 15 seconds. |
N/A | subscription.retries | Optional. The maximum number of subscription retry attempts. This property is only configurable through the Decodable CLI. Defaults to 3. |
Subscription Deadlines
Messages are acknowledged to Pub/Sub approximately every 10 seconds. Best practices are to set the subscription's acknowledgement deadline to be much larger, or else the messages may be processed multiple times.
-
Select the stream that you’d like to connect to this connector. Then, select Next.
-
Define the connection’s schema. Select New Schema to manually enter the fields and field types present or Structured Schema Definition if you want to paste the schema in the form of an Avro or JSON array.
- The stream's schema must match the schema of the data that you plan on sending through this connection.
- For more information about creating a stream or defining the stream schema, see Create a Stream.
-
Select Next when you are finished providing defining the connection’s schema.
-
Give the newly created connection a Name and Description and select Save.
Updated 2 months ago