Data Generator

The Datagen connector can be used as a source to generate sample data based on a specified data.type. Use the Datagen connector when you want to test pipeline SQL statements using random data without setting up connections to external systems.

Overview

Connector namedatagen
Typesource
Delivery guaranteeat most once

Use the Datagen Connector

Follow these steps to use the Datagen Connector.

  1. From the Connections page, select the Datagen Connector and complete the following fields.
UI FieldProperty Name in the Decodable CLIDescription
Data Typedata.typeThe type of sample data that you want this connection to emit. Must be one of the following:

- envoy: Emits records formatted in Envoy's default layout pattern. See Default format pattern in the Envoy documentation for an example of what the emitted record looks like.
- counter: Emits records with counter and time fields. The counter field is a monotonically increasing bigintand the time field is a timestamp encoded as a string.
DelaydelaySpecifies the delay in milliseconds to wait between records. Must be greater than 100.

Defaults to 100.
Value Formatvalue.formatMust be set to JSON.
  1. Select the stream that you’d like to connect to this connector. Then, select Next.
  2. On the "Define this Connection's schema" page, select Next. You cannot modify the schema for the Datagen connector.
  3. Give the newly created connection a Name and Description and select Save.