Decodable logs audit events and metrics about itself in the
_metrics streams. You can use these events and metrics to monitor user activity, health, and the state of your Decodable resources. In general, the
_events stream helps you answer questions like “Who deactivated this pipeline, and when?” It also contains state transition information, which helps you answer questions like “Were there any connectivity issues while this connection was running?” The
_metrics stream can help you answer questions like “Do I have a large enough task size for the throughput that my connection is experiencing?” With the information contained in these two streams, you can get a holistic view of your Decodable system at large.
You can preview these streams in Decodable or use a connector to send these events and metrics into downstream databases or observability systems like ElasticSearch or Datadog. Like other Decodable streams, you can also build pipelines that read data from these streams to perform aggregations or set up trigger alerts based on specific conditions in the data. For more details about the type of information that you can find in the
_metrics stream, see System Events and Metrics for more information.
In addition, you can get a high-level view of the status of Decodable pipelines and connections from their respective Decodable Web pages. You can also view additional details about specific pipelines and connections, such as the amount of data that it is receiving and what is connected to it, by selecting the pipeline or connection name.
The following table describes the various statuses that a connection or pipeline can have and what they mean. You can see the status of a connection or pipeline from the Connections or Pipelines pages in Decodable Web or by running the Decodable CLI command
decodable connections get <connection_id> or
decodable pipelines get <pipeline_id>. Every connection and pipeline has an Actual State and Target State status assigned to it, where Actual State refers to the current state of the resource and Target State refers to the desired end state.
The following table describes all of the statuses that a Decodable resource can have. Note that there are only two statuses that can show up as a Target State: Running and Stopped.
|Running||The resource is active and connected to a Decodable stream. When a connection or pipeline has this status, it is ready to receive, process, or send data.|
|Stopped||The resource is not active. It may be connected to a Decodable stream, but it is not actively sending data through it.|
|Starting||The resource is preparing to enter the running state but is not running yet.|
|Stopping||The resource is preparing to be stopped.|
|Retrying||Something happened with the resource, and Decodable is trying to restart it from the most recent checkpoint. There are many different reasons why this can occur. For example, a source connection could have encountered invalid data or an internal Decodable component was temporarily unavailable. Decodable will try to restore the affected resource until you explicitly tell it to stop.|
An error message with details is shown in the detailed view of the specified resource
|Failed||Something is wrong with the resource and no data is being processed through it. There are many different reasons why this can occur. Manual intervention is required in order to restore it. For assistance, contact [email protected] with the detailed error message shown.|
Every Decodable connection and pipeline measures data flow metrics as it processes data. You can view the Overview page for a specific connection or pipeline to view what streams are connected to it, how many tasks are assigned to it, and how much data is flowing through. Use the Overview page to review the inbound and outbound data metrics of your Decodable resource and confirm whether data is flowing through as expected.
You can also view all available metrics for every active connection and pipeline in the Decodable account by previewing the
_metrics stream. See System Events and Metrics for more information about the available metrics and metric properties in the
To access the Overview page of a specific resource, do the following:
- Navigate to the Connections page or Pipelines page, depending on which resource you want to see metrics for.
- Select the row that represents the resource that you want to inspect. The Overview page for that resource opens.
The Overview page of a Decodable resource displays the following data flow information. These metrics are shown in real-time, so the numbers fluctuate over time.
Metrics are reset upon each activation. Stopping and restarting a connection or pipeline resets metrics back to zero. Metrics are also reset if a connection or pipeline enters a retrying state.
Note: Output metrics for the REST Connector and the DataGen Connector are not supported.
Note: Metrics are not supported for Custom Pipelines.
Datadog is a monitoring and observability platform, enabling you to keep track of all aspects of your Cloud environment in one centralized location. Decodable provides an out-of-the-box Datadog connector, making it even easier for you to monitor your Decodable accounts in your Datadog dashboards. With Decodable's Datadog connector, you can send data from Decodable’s
_metrics stream which is where Decodable writes metrics about the data that it is processing. You can also send metrics from your own custom metrics stream to the Datadog Connector, as long as the metrics stream has a metric name, metric value, and timestamp columns or fields. The Datadog Connector also supports adding tags to the data, either for all metrics, or on a per-metric basis.
See the following pages for more information.
Updated 10 days ago