Decodable generates data about the health, state, and throughput of its resources that is recorded into the _events and _metrics streams. You can use these streams to monitor the user activity and health of Decodable.

Events

Events generated by Decodable include:

Audit events of CRUD operations on your resources (connections, streams, and pipelines)
State transitions of active resources
Errors encountered by your active resources

Events are available in the _events stream and have the following schema.

Name	Type	Description
id	STRING	A unique identifier for the event.
account_id	STRING	The account of the event.
resource_type	STRING	The type of resource that the event relates to. Values are: connection, stream, pipeline, or table.
resource_id	STRING	A unique identifier for the resource the event relates to.
timestamp	BIGINT	The epoch timestamp of the event.
audit_event_type	STRING	The category of audit event.
message	STRING	A human-readable message for the event.
properties	MAP<STRING, STRING>	Additional key-value properties for metadata associated with the audit event type.

A sample event

{
  "id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
  "account_id": "12345678",
  "resource_type": "stream",
  "resource_id": "14c78d3e",
  "timestamp": 1664213138882,
  "audit_event_type": "api_stream_create",
  "message": "Stream created",
  "properties": {
    "user_id": "google-oauth2|102883480266440686107"
  }
}

Metrics

Metrics for active connections and pipelines are available in the _metrics stream and have the following schema.

Name	Type	Description
timestamp	TIMESTAMP(3)	The UTC time that the metric was emitted.
cloud	STRING	The cloud provider where the resource is located (e.g. "aws").
region	STRING	The region where the resource is located (e.g. "us-east-1").
cell	STRING	The Decodable cell where the resource is located. Each Decodable account belongs to one cell.
account_id	STRING	The Decodable account that the resource belongs to.
resource_type	STRING	The type of resource the metric relates to (e.g. "pipeline").
resource_id	STRING	The unique identifier of the resource the metric relates to.
resource_name	STRING	The name of the resource that the metric relates to.
metric_name	STRING	The name of the metric.
metric_value	DOUBLE	The value of the metric. If this value is non-numeric or undefined, then the metric_value is null.
properties	MAP<STRING, STRING>	Key-valued properties for additional metric dimensions. Keys are listed in the "Metric properties" table below.

Metrics are currently not supported for custom pipelines.

Available metrics

The following table describes the metrics that are available in the _metrics stream.

Metric Name	Description
bytes_in_total	The total bytes received.
bytes_in_per_second	The average number of bytes received per second.
bytes_out_total	The total bytes sent.
bytes_out_per_second	The average number of bytes sent per second.
parse_errors_per_second	The average number of events which could not be parsed per second. Applicable to source connections only.
parse_errors_total	The total number of errors which could not be parsed.
records_in_per_second	The average number of events received per second.
records_in_total	The total events received.
records_lag_max	The number of records that are ready for processing, but have not been processed.
records_lag_total	The total number of unconsumed records across all partitions.
records_out_per_second	The average number of events sent per second.
records_out_total	The total events sent.

Metric properties

The following table describes the properties that are available in a metric.

Property Key	Present When	Description
connection_direction	resource_type is connection	Whether the connection is a source or sink.
connector	resource_type is connection	The type of the connector, for example "kafka" or "postgres".
pipeline_version	resource_type is pipeline	The version of the pipeline.
stream_direction	metric relates to I/O against a single stream	Whether the stream is being used as a source or a sink by the resource.
stream_id	metric relates to I/O against a single stream	The id of the stream.
stream_name	metric relates to I/O against a single stream	The name of the stream.
stream_type	metric is attributable to a single stream	The type of the stream. This can be either "append" or "change".
task_index	metric is attributable to a single task	The task number of the task producing the metric. If only one task is used, then this number is "0".

📘
Metrics reset on connection or pipeline restarts
All of the *_totalmetrics (such as bytes_in_total) reset each time a connection or pipeline restarts. This means you should not use these metrics to compare the total number of records or bytes processed with external systems or across Decodable resources.

A sample metric

{
  "account_id": "00000000",
  "resource_type": "pipeline",
  "resource_id": "b672bf30",
  "resource_name": "pipeline_name",
  "metric_name": "bytes_in_per_second",
  "metric_value": 83.00278357689632,
  "timestamp": "2023-01-25 23:26:46.816",
  "cloud": "aws",
  "region": "us-east-1",
  "cell": "2",
  "properties": {
    "pipeline_version": "1",
    "stream_name": "envoy_raw",
    "stream_id": "1c724825",
    "task_index": "0",
    "stream_direction": "source",
    "stream_type": "append"
  }
}