System Events and Metrics

Decodable provides audit events and metrics via pre-provisioned streams in your account. You can view these events and metrics in the stream, or you can use any of Decodable's connectors to import them into downstream observability systems. You can even build pipelines that read from these streams and perform aggregations or trigger on certain conditions for alerting.

Events

Events generated by Decodable include:

  • Audit events of CRUD operations on your resources (connections, streams, and pipelines)
  • State transitions of active resources
  • Errors encountered by your active resources

Events are available in the _events stream and have the following schema:

NameTypeDescription
idSTRINGA unique identifier for the event
account_idSTRINGThe account of the event
resource_typeSTRINGThe type of resource the event relates to. Values are: connection, stream, pipeline, or table
resource_idSTRINGThe unique identifier of the resource the event relates to
timestampBIGINTThe epoch timestamp of the event
audit_event_typeSTRINGThe category of audit event
messageSTRINGA human-readable message for the event
propertiesMAP<STRING, STRING>Key-valued properties for additional information specific to the audit event type

A sample event:

{
  "id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
  "account_id": "12345678",
  "resource_type": "stream",
  "resource_id": "14c78d3e",
  "timestamp": 1664213138882,
  "audit_event_type": "api_stream_create",
  "message": "Stream created",
  "properties": {
    "user_id": "google-oauth2|102883480266440686107"
  }
}

Metrics

Metrics generated by Decodable include:

  • Throughput metrics on connections
  • Throughput metrics on pipelines, by stream

Metrics are available in the _metrics stream and have the following schema:

NameTypeDescription
idSTRINGA unique identifier for the metric
account_idSTRINGThe account of the metric
resource_typeSTRINGThe type of resource the metric relates to. Values are: connection, stream, pipeline, or table
resource_idSTRINGThe unique identifier of the resource the metric relates to
timestampBIGINTThe epoch timestamp of the metric
metric_nameSTRINGThe name of the metric
metric_valueDOUBLEThe value of the metric
propertiesMAP<STRING, STRING>Key-valued properties for additional metric dimensions

Available metrics:

Metric NameDescription
decodable.taskmanager.task.numBytesInTotal bytes received
decodable.taskmanager.task.numBytesInPerSecond.rateMoving average bytes received per second
decodable.taskmanager.task.numBytesOutTotal bytes sent
decodable.taskmanager.task.numBytesOutPerSecond.rateMoving average bytes sent per second
decodable.taskmanager.task.numRecordsInTotal events received
decodable.taskmanager.task.numRecordsInPerSecond.rateMoving average events received per second
decodable.taskmanager.task.numRecordsOutTotal events sent
decodable.taskmanager.task.numRecordsOutPerSecond.rateMoving average events sent per second

A sample metric:

{
  "id": "9723345d-1d4b-47e2-b7e6-f27d177bba02",
  "account_id": "00000000",
  "resource_type": "pipeline",
  "resource_id": "fa0488ce",
  "timestamp": 1664216738213,
  "metric_name": "decodable.taskmanager.task.numBytesOut",
  "metric_value": 15820.12,
  "properties": {
    "stream_name": "envoy_processed",
    "is_source": "true",
    "task_id": "bc764cd8ddf7a0cff126f51c16239658",
    "subtask_index": "0"
  }
}