Edit or delete pipelines

You can view, edit, and delete pipelines from the Pipelines page in your account. Select a pipeline to view information such as the number of records that the pipeline is processing, which streams the pipeline is connected to, and the SQL statement that it is using.

Edit a pipeline

You can modify the SQL of a pipeline by completing the following steps.

  1. Navigate to the Pipelines page.

  2. Select the pipeline that you want to edit.

  3. Select SQL and modify the pipeline as needed. When you are finished, select Save as New Version. When you save changes you’ve made to a pipeline, it’ll appear as a new version under the Version tab. You can select any version to view it.

    1. Remember that the schema of the pipeline must match the schema of the destination stream. If you modify your pipeline so that the schema of the data being processed by it does not match the schema that the output stream expects, then you will be unable to activate the pipeline.

  4. Activate the latest version of the pipeline by selecting the Version tab, selecting the latest version, and selecting Start. When you activate a pipeline, you can specify the following optional parameters:

    1. Task Size: The amount of resources, such as how much CPU or Memory, to allocate for streaming and processing data. See Manage Task Count and Sizes for more information on managing task count and sizes.

    2. Task Count: The number of pipeline workers that perform data collection or processing. See Manage Task Count and Sizes for more information on managing task count and sizes.

    3. Discard State: Choose whether you want to discard state. By selecting this option, you are reactivating the pipeline without attempting to restore state from a save point or checkpoint. This means that once the pipeline is activated, data ingestion resumes based on the Earliest / Latest value.

      1. NOTE: If you want to start data ingestion from the earliest offset, you must select this option.

    4. Earliest / Latest: Specify where in the stream that a pipeline should start ingesting data from. Select Earliest to start processing data from the earliest record available or Latest to start processing data from the latest record available.

Delete a pipeline

You can delete a pipeline by completing the following steps.

Prerequisites

The pipeline must not be running. Make sure to stop the pipeline before deleting it.

Steps

  1. Navigate to the Pipelines page.

  2. In the row that lists your pipeline, select the More Options (…​) icon and select Delete.

  3. When prompted to confirm your choice, select Delete.