Integrate with GitHub Actions

The Decodable CLI can be used in GitHub Actions to automate deployment of Decodable resources, such as pipelines and connections.

At a high level, the workflow should do the following:

  1. Generate an access token

  2. Use the Decodable CLI to apply Decodable resource definitions

Below is an example of doing this with a custom Flink pipeline.

Prerequisites

Authentication

To use the Decodable CLI it needs to have an access token for authentication. This token has a limited lifespan and needs to be refreshed periodically. When using the Decodable CLI in a GitHub Actions workflow the token is generated from a client_id and client_secret pair that are unique to your Decodable account. To get the credential pair for your account, contact Decodable.

Resource definition

Declarative resource management is a powerful feature of Decodable to define one or more Decodable resources using YAML. These can be written by hand, or more commonly, extracted from existing resources.

Here is an example resource definition to create a custom Flink pipeline written in Java.

resources.yaml
---
kind: pipeline
metadata:
    name: java_pipeline
spec_version: v1
spec:
    type: JAVA
    job_file_path: target/decodable-custom-pipeline-0.1.jar
    properties:
        flink_version: 1.18-java11

Generating an access token

An access token is required for the Decodable CLI to authenticate as your account. The token can be generated using the client ID and secret and a REST call to the authentication service.

Once retrieved, the token needs combining with an expiry date (1 day from the time at which it was fetched), and writing to the .decodable/auth file that the Decodable CLI uses.

Here is an example of a shell script that does this.

store-auth.sh
response=$(curl -X POST https://auth.decodable.co/oauth/token      \
                -H 'Content-Type: application/json; charset=utf-8' \
                --data-binary @- <<EOF
                {
                    "client_id":      "$DECODABLE_CLI_CLIENT_ID",
                    "client_secret":  "$DECODABLE_CLI_CLIENT_SECRET",
                    "audience":       "https://api.decodable.co/",
                    "grant_type":     "client_credentials"
                }
                EOF
)

access_token=$(echo $response | jq -r .access_token)
expiry=$(date -d "+1 day" +"%Y-%m-%dT%H:%M:%S.%6N%:z")

cat <<EOT > ~/.decodable/auth
version: 1.0.0
tokens:
    default:
        access_token: $access_token
        expiry: $expiry
EOT

It’s assumed that you have set your client ID and secret as environment variables DECODABLE_CLI_CLIENT_ID and DECODABLE_CLI_CLIENT_SECRET respectively.

Other OS’s may have different versions of tools such as date, and require different invocation.

Applying the resources using Decodable CLI

Once the authentication file is created you can call the Decodable CLI to apply the resources to your Decodable account:

apply-resources.sh
decodable apply resources.yaml

Putting it together in a GitHub workflow

This example GitHub Actions workflow will run when a pull request is merged. It will create the access token needed for the Decodable CLI, and then invoke the CLI to apply the resource definition specified in resource.yaml.

To use this workflow you should store your client ID and secret (that you obtained in the prerequisites section above) as GitHub secrets. These are then set as environment variables in the workflow for use in the store-auth.sh script.

.github/workflows/deploy-decodable.yaml
name: Decodable Deployment
on:
  pull_request:
    types:
      - closed

jobs:
  apply_decodable_resources:
    if: github.event.pull_request.merged == true
    name: Apply Decodable Resources
    runs-on: ubuntu-22.04
    steps:
      - name: Generate the authentication token
        env:
          DECODABLE_CLI_CLIENT_ID:     ${{ secrets.DECODABLE_CLI_CLIENT_ID }}
          DECODABLE_CLI_CLIENT_SECRET: ${{ secrets.DECODABLE_CLI_CLIENT_SECRET }}
        run: ./store-auth.sh
      - name: Apply the Decodable resources
        run: ./apply-resources.sh