As we discussed in the Pipelines concept page, Decodable pipelines use SQL to process data. Pipeline SQL should feel familiar to anyone who has used relational/batch database systems. The primary differences you'll notice are that:
- All pipelines run a single SQL statement in the form
INSERT INTO <sink_stream> SELECT ... FROM <source_stream> ...
- Pipeline queries specify at least one source and exactly one sink stream. See Pipeline Examples for examples of pipeline queries.
- Decodable SQL is a subset of the Flink SQL standard with some additional functionality. See Function Reference for a detailed list of functions available for use by your pipeline queries.
- Windows allow us to aggregate over periods of time. See Windowing to learn more about this key streaming concept.
- You activate a pipeline to start it, and deactivate a pipeline to stop it.
Updated 7 months ago