Manage state with snapshots

Decodable will automatically manage state for your pipelines when you deactivate and reactivate them. However, if you want to manage state in a more fine-grained manner you can use snapshots to manage the state of your pipelines.

A snapshot is a point-in-time, consistent image of the internal state of your job. They contain information necessary to resume processing from that point in time, such as source offsets and intermediate results of joins. Snapshots can be generated on a schedule or on demand. Snapshots are associated with a single pipeline and are retained for seven days by default.

Snapshots are managed through the Decodable CLI, primarily with the decodable pipeline snapshot subcommand.

Creating snapshots

One-time snapshots

To create a one-time snapshots on demand use decodable pipeline snapshot create.

For example, to take a one-off snapshot of pipeline f79ebea0:

$ decodable pipeline snapshot create f79ebea0
Created snapshot One-time snapshot for [f79ebea0] at [2024-01-01T16:50:00.000000000Z] (40479f00)

The id of the created snapshot is shown in the output of the command (in this example, 40479f00).

To override the default retention period of seven days, pass the --retention parameter and supply the desired value expressed in seconds.

Scheduled snapshots

To configure Decodable to automatically take snapshots of a running pipeline update the configuration of the pipeline, use decodable pipeline update and add the following parameters:

--snapshots
--snapshots-cron. This takes a standard cron syntax.

You can generate cron expressions using this online tool.

For example, to configure pipeline f79ebea0 to take a snapshot every 15 minutes:

$ decodable pipeline update f79ebea0 --snapshots --snapshots-cron "*/15 * * * *"
Updated pipeline "f79ebea0"

To override the default retention period of seven days, pass the --snapshots-retention parameter and supply the desired value expressed in seconds.

Using snapshots

You can use any completed, non-expired snapshot to reset pipeline state to the point in time the snapshot represents.

For example, to restore pipeline f79ebea0 to the snapshot 40479f00:

$ decodable pipeline activate f79ebea0 --start-from-snapshot-id 40479f00
id         version    target state
f79ebea0   2          RUNNING

Restoring a pipeline to a snapshot overwrites the current state of your pipeline with that of the snapshot. Restoring to a snapshot can cause data to be delivered multiple times, violating exactly-once delivery guarantees.

Source data must still be available in order to be reprocessed.

Viewing snapshot details

To view a list of all snapshots for a given pipeline use the list subcommand.

This lists all snapshots for pipeline f79ebea0:

$ decodable pipeline snapshot list f79ebea0
id        name                                                                   state     start time
40479f00  One-time snapshot for [f79ebea0] at [2024-01-01T16:50:00.000000000Z]   FINISHED  2024-01-01T16:50:00Z
24a9b155  Scheduled snapshot for [f79ebea0] at [2024-01-01T16:45:00.000000000Z]  FINISHED  2024-01-01T16:45:00Z
a261bac9  Scheduled snapshot for [f79ebea0] at [2024-01-01T16:30:00.000000000Z]  FINISHED  2024-01-01T16:30:00Z
07ca95c4  Scheduled snapshot for [f79ebea0] at [2024-01-01T16:15:00.000000000Z]  FINISHED  2024-01-01T16:15:00Z
1bfbe7f1  Scheduled snapshot for [f79ebea0] at [2024-01-01T16:00:00.000000000Z]  FINISHED  2024-01-01T16:00:00Z

To view the status and details of a particular snapshot use the get subcommand.

This shows the details for snapshot 40479f00 associated with pipeline f79ebea0:

$ decodable pipeline snapshot get f79ebea0 40479f00
One-time snapshot for [f79ebea0] at [2024-01-01T16:50:00.000000000Z]
  id                       40479f00
  pipeline id              f79ebea0
  type                     SCHEDULED
  start time               2024-01-01T16:50:00Z
  end time                 2024-01-01T16:50:01Z
  retention time           2024-01-08T16:50:01Z
  state                    FINISHED

Removing snapshots

Snapshots expire after seven days. This period can be overridden when the snapshot is created. To remove a snapshot sooner use the delete subcommand:

$ decodable pipeline snapshot delete f79ebea0 40479f00
Deleted snapshot 40479f00