StarTree Cloud provides managed hosting for Apache Pinot on all major cloud platforms, including AWS, GCP and Azure. StarTree Cloud enables developers to provision Apache Pinot clusters of different sizes, ingest data from real-time and batch data sources, and run analytics workloads with ultra-low latency. Meanwhile, StarTree manages the underlying infrastructure for you, allowing you to have timely insights from a diverse set of data and make informed business decisions. StarTree Cloud consists of tools that give you a better developer experience, including StarTree Dataset Manager (for data ingestion) and StarTree ThirdEye (for Anomaly Detection).
Sending a Decodable data stream to StarTree is accomplished in two stages, first by creating a sink connector to a data source that is supported by StarTree, and then by adding that data source to your StarTree configuration. Decodable and StarTree mutually support several technologies, including the following:
- Amazon S3
- Apache Kafka
This example demonstrates using Kafka as the sink from Decodable and the source for StarTree. Sign in to the Decodable Web Console and follow the configuration steps provided for the Kafka Connector to create a
sink connector. For examples of using the command line tools or scripting, see the How To guides.
The StarTree Data Manager can ingest messages from your own Kafka cluster or hosted Kafka services like Confluent Cloud or Amazon MSK.
Create a dataset.
For the data source, select Streaming and then Kafka.
For the data modeling, select the topic that you'd like to connect to Pinot and its format.
Optionally, add a primary key to enable upserts to your real-time table.
Review dataset configuration and submit to create.
For more detailed information, please refer to StarTree's Kafka documentation.
Updated 10 months ago