Decodable provides audit events and metrics via pre-provisioned streams in your account. You can view these events and metrics in the stream, or you can use any of Decodable's connectors to import them into downstream observability systems. You can even build pipelines that read from these streams and perform aggregations or trigger on certain conditions for alerting.

Events

Events generated by Decodable include:

Audit events of CRUD operations on your resources (connections, streams, and pipelines)
State transitions of active resources
Errors encountered by your active resources

Events are available in the _events stream and have the following schema:

Name	Type	Description
id	STRING	A unique identifier for the event
account_id	STRING	The account of the event
resource_type	STRING	The type of resource the event relates to. Values are: connection, stream, pipeline, or table
resource_id	STRING	The unique identifier of the resource the event relates to
timestamp	BIGINT	The epoch timestamp of the event
audit_event_type	STRING	The category of audit event
message	STRING	A human-readable message for the event
properties	MAP<STRING, STRING>	Key-valued properties for additional information specific to the audit event type

A sample event:

{
  "id": "f6a77bdb-eb41-4a18-b300-05450194dd0b",
  "account_id": "12345678",
  "resource_type": "stream",
  "resource_id": "14c78d3e",
  "timestamp": 1664213138882,
  "audit_event_type": "api_stream_create",
  "message": "Stream created",
  "properties": {
    "user_id": "google-oauth2|102883480266440686107"
  }
}

Metrics

Metrics for active connections and pipelines are available in the _metrics stream and have the following schema:

Name	Type	Description
timestamp	TIMESTAMP(3)	The UTC time that the metric was emitted
cloud	STRING	The cloud provider where the resource is located (e.g. "aws")
region	STRING	The region where the resource is located (e.g. "us-east-1")
cell	STRING	The Decodable cell where the resource is located. Each Decodable account belongs to one and only one cell.
account_id	STRING	The Decodable account that the resource belongs to
resource_type	STRING	The type of resource the metric relates to (e.g. "pipeline")
resource_id	STRING	The unique identifier of the resource the metric relates to
resource_name	STRING	The name of the resource that the metric relates to
metric_name	STRING	The name of the metric
metric_value	DOUBLE	The value of the metric (may be null if non-numeric or undefined)
properties	MAP<STRING, STRING>	Key-valued properties for additional metric dimensions. Keys are listed in a table below.

Available metrics:

Metric Name	Description
bytes_in_total	Total bytes received since the last activation of the pipeline/connection.
bytes_in_per_second	Rolling average number of bytes received per second
bytes_out_total	Total bytes sent since the last activation of the pipeline/connection.
bytes_out_per_second	Rolling average number of bytes sent per second
records_in_total	Total events received since the last activation of the pipeline/connection.
records_in_per_second	Rolling average number of events received per second
records_out_total	Total events sent since the last activation of the pipeline/connection.
records_out_per_second	Rolling average number of events sent per second
parse_errors_total	Total number of errors which could not be parsed
parse_errors_per_second	Rolling average number of events which could not be parsed per second (source connections only)

Metric properties:

Property Key	Present When	Description
connection_direction	resource_type is connection	Whether the connection is a source or sink
connector	resource_type is connection	The type of the connector (e.g. "kafka" or "postgres")
pipeline_version	resource_type is pipeline	The version of the pipeline
stream_direction	metric relates to I/O against a single stream	Whether the stream is being used as a source or a sink by the resource
stream_id	metric relates to I/O against a single stream	The id of the stream
stream_name	metric relates to I/O against a single stream	The name of the stream
stream_type	metric is attributable to a single stream	The type of the stream (e.g. "append" or "change")
task_index	metric is attributable to a single task	The task number of the task producing the metric. When only one task is used, this is always "0".

A sample metric:

{
  "account_id": "00000000",
  "resource_type": "pipeline",
  "resource_id": "b672bf30",
  "resource_name": "pipeline_name",
  "metric_name": "bytes_in_per_second",
  "metric_value": 83.00278357689632,
  "timestamp": "2023-01-25 23:26:46.816",
  "cloud": "aws",
  "region": "us-east-1",
  "cell": "2",
  "properties": {
    "pipeline_version": "1",
    "stream_name": "envoy_raw",
    "stream_id": "1c724825",
    "task_index": "0",
    "stream_direction": "source",
    "stream_type": "append"
  }
}