Integrate with GitHub Actions

The Decodable CLI can be used in GitHub Actions to automate deployment of Decodable resources, such as pipelines and connections.

At a high level, the workflow uses the Decodable CLI to apply Decodable resource definitions. See this guide for details of invoking the CLI from a script.

Below is an example of doing this with a custom Flink pipeline.

Prerequisites

Authentication

It’s recommended that you create a dedicated user within your Decodable account with the permissions required. The benefit of a dedicated account is that any personnel changes at your company won’t impact the automation that you create here.

Storing authentication details

Run decodable login --file my_account.auth and follow the prompts to login to Decodable using the dedicated account.
Extract the value for refresh token: from my_account.auth. You can do this programmatically with yq:
```
yq '.tokens.default.refresh_token' my_account.auth
```
Store the refresh token value as a GitHub secret. In the example below we’ll assume that it’s stored as a secret called DECODABLE_REFRESH_TOKEN.

Resource definition

Declarative resource management is a powerful feature of Decodable to define one or more Decodable resources using YAML. These can be written by hand, or more commonly, extracted from existing resources.

Here is an example resource definition to create a custom Flink pipeline written in Java.

resources.yaml

---
kind: pipeline
metadata:
    name: java_pipeline
spec_version: v1
spec:
    type: JAVA
    job_file_path: target/decodable-custom-pipeline-0.1.jar
    properties:
        flink_version: 1.18-java11

Suggested workflow

Install CLI

You will need to make the CLI available in your GitHub workflow. You can download the CLI from here.

An example Docker container for the CLI is also available for reference.

Configure CLI

As described here, two files must be present for the CLI to work without user interaction. Below is an example of generating these.

Authentication details

cat <<EOF > ~/.decodable/auth
version: 1.0.0
tokens:
    default:
        refresh_token: ${{ secrets.DECODABLE_REFRESH_TOKEN }}
EOF

Account details

cat <<EOF > ~/.decodable/config
version: 1.0.0
active-profile: default

profiles:
  default:
    account: ${{ vars.DECODABLE_ACCOUNT_NAME }}
EOF

This assumes that you’ve stored your Decodable account name in a GitHub variable called $DECODABLE_ACCOUNT_NAME

Apply the Decodable resources

Once the authentication file is created, you can call the Decodable CLI to apply the resources to your Decodable account:

decodable apply resources.yaml

Putting it together in a GitHub workflow

This example GitHub Actions workflow will run when code is pushed to the main branch.

The workflow installs the CLI, configures it, and then applies all the resource definitions under the folder decodable-cicd. For example, you might have the following structure:

$ tree declarative-cicd
declarative-cicd
├── connections
│   └── datagen-01.yaml
└── pipelines
    └── count-messages.yaml

To use this workflow you should store:

DECODABLE_ACCOUNT_NAME GitHub repository variable
DECODABLE_REFRESH_TOKEN GitHub secret

.github/workflows/deploy-decodable.yaml

name: Deploy Decodable resources

on:
  push:
    branches:
      - main
  # pull_request:
  workflow_dispatch:

jobs:
  setup-and-apply:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Download and unpack the Decodable CLI
      run: |
        curl -L https://releases.decodable.co/decodable-cli/linux/amd64/decodable-cli-linux-amd64-1.20.0.tar.gz -o /tmp/decodable-cli.tar.gz && \
        tar -xzf /tmp/decodable-cli.tar.gz -C /tmp && \
        sudo mv /tmp/decodable-cli-linux-amd64-1.20.0/bin/decodable /usr/local/bin && \
        rm -r /tmp/decodable-cli.tar.gz /tmp/decodable-cli-linux-amd64-1.20.0

    - name: Create Decodable CLI auth file
      run: |
        mkdir -p ~/.decodable
        cat <<EOF > ~/.decodable/auth
        version: 1.0.0
        tokens:
          default:
            refresh_token: ${{ secrets.DECODABLE_REFRESH_TOKEN }}
        EOF

    - name: Create Decodable CLI config file
      run: |
        mkdir -p ~/.decodable
        cat <<EOF > ~/.decodable/config
        version: 1.0.0
        active-profile: default

        profiles:
          default:
            account: ${{ vars.DECODABLE_ACCOUNT_NAME }}
        EOF

    - name: Apply all Decodable resources
      run: decodable apply $(find declarative-cicd -name '*.yaml')