System Events and Metrics
Decodable generates data about the health, state, and throughput of its resources that is recorded into the _events
and _metrics
streams. You can use these streams to monitor the user activity and health of Decodable.
Events
Events generated by Decodable include:
- Audit events of CRUD operations on your resources (connections, streams, and pipelines)
- State transitions of active resources
- Errors encountered by your active resources
Events are available in the _events
stream and have the following schema.
Name | Type | Description |
---|---|---|
id | STRING | A unique identifier for the event. |
account_id | STRING | The account of the event. |
resource_type | STRING | The type of resource that the event relates to. Values are: connection, stream, pipeline, or table. |
resource_id | STRING | A unique identifier for the resource the event relates to. |
timestamp | BIGINT | The epoch timestamp of the event. |
audit_event_type | STRING | The category of audit event. |
message | STRING | A human-readable message for the event. |
properties | MAP<STRING, STRING> | Additional key-value properties for metadata associated with the audit event type. |
A sample event
{
"id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
"account_id": "12345678",
"resource_type": "stream",
"resource_id": "14c78d3e",
"timestamp": 1664213138882,
"audit_event_type": "api_stream_create",
"message": "Stream created",
"properties": {
"user_id": "google-oauth2|102883480266440686107"
}
}
Metrics
Metrics for active connections and pipelines are available in the _metrics
stream and have the following schema.
Name | Type | Description |
---|---|---|
timestamp | TIMESTAMP(3) | The UTC time that the metric was emitted. |
cloud | STRING | The cloud provider where the resource is located (e.g. "aws"). |
region | STRING | The region where the resource is located (e.g. "us-east-1"). |
cell | STRING | The Decodable cell where the resource is located. Each Decodable account belongs to one cell. |
account_id | STRING | The Decodable account that the resource belongs to. |
resource_type | STRING | The type of resource the metric relates to (e.g. "pipeline"). |
resource_id | STRING | The unique identifier of the resource the metric relates to. |
resource_name | STRING | The name of the resource that the metric relates to. |
metric_name | STRING | The name of the metric. |
metric_value | DOUBLE | The value of the metric. If this value is non-numeric or undefined, then the metric_value is null. |
properties | MAP<STRING, STRING> | Key-valued properties for additional metric dimensions. Keys are listed in the "Metric properties" table below. |
Metrics are currently not supported for custom pipelines.
Available metrics
The following table describes the metrics that are available in the _metrics
stream.
Metric Name | Description |
---|---|
bytes_in_total | The total bytes received. |
bytes_in_per_second | The average number of bytes received per second. |
bytes_out_total | The total bytes sent. |
bytes_out_per_second | The average number of bytes sent per second. |
parse_errors_per_second | The average number of events which could not be parsed per second. Applicable to source connections only. |
parse_errors_total | The total number of errors which could not be parsed. |
records_in_per_second | The average number of events received per second. |
records_in_total | The total events received. |
records_lag_max | The number of records that are ready for processing, but have not been processed. |
records_lag_total | The total number of unconsumed records across all partitions. |
records_out_per_second | The average number of events sent per second. |
records_out_total | The total events sent. |
Metric properties
The following table describes the properties that are available in a metric.
Property Key | Present When | Description |
---|---|---|
connection_direction | resource_type is connection | Whether the connection is a source or sink. |
connector | resource_type is connection | The type of the connector, for example "kafka" or "postgres". |
pipeline_version | resource_type is pipeline | The version of the pipeline. |
stream_direction | metric relates to I/O against a single stream | Whether the stream is being used as a source or a sink by the resource. |
stream_id | metric relates to I/O against a single stream | The id of the stream. |
stream_name | metric relates to I/O against a single stream | The name of the stream. |
stream_type | metric is attributable to a single stream | The type of the stream. This can be either "append" or "change". |
task_index | metric is attributable to a single task | The task number of the task producing the metric. If only one task is used, then this number is "0". |
Metrics reset on connection or pipeline restarts
All of the
*_total
metrics (such asbytes_in_total
) reset each time a connection or pipeline restarts. This means you should not use these metrics to compare the total number of records or bytes processed with external systems or across Decodable resources.
A sample metric
{
"account_id": "00000000",
"resource_type": "pipeline",
"resource_id": "b672bf30",
"resource_name": "pipeline_name",
"metric_name": "bytes_in_per_second",
"metric_value": 83.00278357689632,
"timestamp": "2023-01-25 23:26:46.816",
"cloud": "aws",
"region": "us-east-1",
"cell": "2",
"properties": {
"pipeline_version": "1",
"stream_name": "envoy_raw",
"stream_id": "1c724825",
"task_index": "0",
"stream_direction": "source",
"stream_type": "append"
}
}
Updated 4 months ago