System Events and Metrics
Decodable provides audit events and metrics via pre-provisioned streams in your account. You can view these events and metrics in the stream, or you can use any of Decodable's connectors to import them into downstream observability systems. You can even build pipelines that read from these streams and perform aggregations or trigger on certain conditions for alerting.
Events
Events generated by Decodable include:
- Audit events of CRUD operations on your resources (connections, streams, and pipelines)
- State transitions of active resources
- Errors encountered by your active resources
Events are available in the _events stream and have the following schema:
Name | Type | Description |
---|---|---|
id | STRING | A unique identifier for the event |
account_id | STRING | The account of the event |
resource_type | STRING | The type of resource the event relates to. Values are: connection, stream, pipeline, or table |
resource_id | STRING | The unique identifier of the resource the event relates to |
timestamp | BIGINT | The epoch timestamp of the event |
audit_event_type | STRING | The category of audit event |
message | STRING | A human-readable message for the event |
properties | MAP<STRING, STRING> | Key-valued properties for additional information specific to the audit event type |
A sample event:
{
"id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
"account_id": "12345678",
"resource_type": "stream",
"resource_id": "14c78d3e",
"timestamp": 1664213138882,
"audit_event_type": "api_stream_create",
"message": "Stream created",
"properties": {
"user_id": "google-oauth2|102883480266440686107"
}
}
Metrics
Metrics for active connections and pipelines are available in the _metrics stream and have the following schema:
Name | Type | Description |
---|---|---|
timestamp | TIMESTAMP(3) | The UTC time that the metric was emitted |
cloud | STRING | The cloud provider where the resource is located (e.g. "aws") |
region | STRING | The region where the resource is located (e.g. "us-east-1") |
cell | STRING | The Decodable cell where the resource is located. Each Decodable account belongs to one and only one cell. |
account_id | STRING | The Decodable account that the resource belongs to |
resource_type | STRING | The type of resource the metric relates to (e.g. "pipeline") |
resource_id | STRING | The unique identifier of the resource the metric relates to |
resource_name | STRING | The name of the resource that the metric relates to |
metric_name | STRING | The name of the metric |
metric_value | DOUBLE | The value of the metric (may be null if non-numeric or undefined) |
properties | MAP<STRING, STRING> | Key-valued properties for additional metric dimensions. Keys are listed in a table below. |
Available metrics:
Metric Name | Description |
---|---|
bytes_in_total | Total bytes received since the last activation of the pipeline/connection. |
bytes_in_per_second | Rolling average number of bytes received per second |
bytes_out_total | Total bytes sent since the last activation of the pipeline/connection. |
bytes_out_per_second | Rolling average number of bytes sent per second |
records_in_total | Total events received since the last activation of the pipeline/connection. |
records_in_per_second | Rolling average number of events received per second |
records_out_total | Total events sent since the last activation of the pipeline/connection. |
records_out_per_second | Rolling average number of events sent per second |
parse_errors_total | Total number of errors which could not be parsed |
parse_errors_per_second | Rolling average number of events which could not be parsed per second (source connections only) |
Metric properties:
Property Key | Present When | Description |
---|---|---|
connection_direction | resource_type is connection | Whether the connection is a source or sink |
connector | resource_type is connection | The type of the connector (e.g. "kafka" or "postgres") |
pipeline_version | resource_type is pipeline | The version of the pipeline |
stream_direction | metric relates to I/O against a single stream | Whether the stream is being used as a source or a sink by the resource |
stream_id | metric relates to I/O against a single stream | The id of the stream |
stream_name | metric relates to I/O against a single stream | The name of the stream |
stream_type | metric is attributable to a single stream | The type of the stream (e.g. "append" or "change") |
task_index | metric is attributable to a single task | The task number of the task producing the metric. When only one task is used, this is always "0". |
A sample metric:
{
"account_id": "00000000",
"resource_type": "pipeline",
"resource_id": "b672bf30",
"resource_name": "pipeline_name",
"metric_name": "bytes_in_per_second",
"metric_value": 83.00278357689632,
"timestamp": "2023-01-25 23:26:46.816",
"cloud": "aws",
"region": "us-east-1",
"cell": "2",
"properties": {
"pipeline_version": "1",
"stream_name": "envoy_raw",
"stream_id": "1c724825",
"task_index": "0",
"stream_direction": "source",
"stream_type": "append"
}
}
Updated about 1 month ago