System Events and Metrics

Decodable provides audit events and metrics via pre-provisioned streams in your account. You can view these events and metrics in the stream, or you can use any of Decodable's connectors to import them into downstream observability systems. You can even build pipelines that read from these streams and perform aggregations or trigger on certain conditions for alerting.

Events

Events generated by Decodable include:

  • Audit events of CRUD operations on your resources (connections, streams, and pipelines)
  • State transitions of active resources
  • Errors encountered by your active resources

Events are available in the _events stream and have the following schema:

NameTypeDescription
idSTRINGA unique identifier for the event
account_idSTRINGThe account of the event
resource_typeSTRINGThe type of resource the event relates to. Values are: connection, stream, pipeline, or table
resource_idSTRINGThe unique identifier of the resource the event relates to
timestampBIGINTThe epoch timestamp of the event
audit_event_typeSTRINGThe category of audit event
messageSTRINGA human-readable message for the event
propertiesMAP<STRING, STRING>Key-valued properties for additional information specific to the audit event type

A sample event:

{
  "id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
  "account_id": "12345678",
  "resource_type": "stream",
  "resource_id": "14c78d3e",
  "timestamp": 1664213138882,
  "audit_event_type": "api_stream_create",
  "message": "Stream created",
  "properties": {
    "user_id": "google-oauth2|102883480266440686107"
  }
}

Metrics

Metrics for active connections and pipelines are available in the _metrics stream and have the following schema:

NameTypeDescription
timestampTIMESTAMP(3)The UTC time that the metric was emitted
cloudSTRINGThe cloud provider where the resource is located (e.g. "aws")
regionSTRINGThe region where the resource is located (e.g. "us-east-1")
cellSTRINGThe Decodable cell where the resource is located. Each Decodable account belongs to one and only one cell.
account_idSTRINGThe Decodable account that the resource belongs to
resource_typeSTRINGThe type of resource the metric relates to (e.g. "pipeline")
resource_idSTRINGThe unique identifier of the resource the metric relates to
resource_nameSTRINGThe name of the resource that the metric relates to
metric_nameSTRINGThe name of the metric
metric_valueDOUBLEThe value of the metric (may be null if non-numeric or undefined)
propertiesMAP<STRING, STRING>Key-valued properties for additional metric dimensions. Keys are listed in a table below.

Available metrics:

Metric NameDescription
bytes_in_totalTotal bytes received since the last activation of the pipeline/connection.
bytes_in_per_secondRolling average number of bytes received per second
bytes_out_totalTotal bytes sent since the last activation of the pipeline/connection.
bytes_out_per_secondRolling average number of bytes sent per second
records_in_totalTotal events received since the last activation of the pipeline/connection.
records_in_per_secondRolling average number of events received per second
records_out_totalTotal events sent since the last activation of the pipeline/connection.
records_out_per_secondRolling average number of events sent per second
parse_errors_totalTotal number of errors which could not be parsed
parse_errors_per_secondRolling average number of events which could not be parsed per second (source connections only)

Metric properties:

Property KeyPresent WhenDescription
connection_directionresource_type is connectionWhether the connection is a source or sink
connectorresource_type is connectionThe type of the connector (e.g. "kafka" or "postgres")
pipeline_versionresource_type is pipelineThe version of the pipeline
stream_directionmetric relates to I/O against a single streamWhether the stream is being used as a source or a sink by the resource
stream_idmetric relates to I/O against a single streamThe id of the stream
stream_namemetric relates to I/O against a single streamThe name of the stream
stream_typemetric is attributable to a single streamThe type of the stream (e.g. "append" or "change")
task_indexmetric is attributable to a single taskThe task number of the task producing the metric. When only one task is used, this is always "0".

A sample metric:

{
  "account_id": "00000000",
  "resource_type": "pipeline",
  "resource_id": "b672bf30",
  "resource_name": "pipeline_name",
  "metric_name": "bytes_in_per_second",
  "metric_value": 83.00278357689632,
  "timestamp": "2023-01-25 23:26:46.816",
  "cloud": "aws",
  "region": "us-east-1",
  "cell": "2",
  "properties": {
    "pipeline_version": "1",
    "stream_name": "envoy_raw",
    "stream_id": "1c724825",
    "task_index": "0",
    "stream_direction": "source",
    "stream_type": "append"
  }
}