Azure powered real time analytics architecture is now more powerful with native integration of Azure Data Explorer as output for Azure Stream Analytics. With this new integration Azure Stream Analytics job can natively ingest the data into Azure Data Explorer table.
This pattern allows low latency on the alerts based on Azure Stream Analytics input stream and with logs being ingested into Azure Data Explorer, you can run deeper investigations, find out patterns, review anomalies with your data and more.
Create Azure Data Explorer output connection to Azure Stream Analytics job
- Under the Azure Stream Analytics job, select Outputs and then select Azure Data Explorer
- Enter the required information into the output connection
- Save your connection and start your Azure Stream Analytics job. Output of your SQL query specified in your Azure Stream Analytics job will now be ingested into your Azure Data Explorer table.
Note: For the ingestion to successfully work, you need to make sure that -
- Number of columns in Azure Stream Analytics job query should match with Azure Data Explorer table and should be in the same order.
- Name of the columns & data type should match between Azure Stream Analytics SQL query and Azure Data Explorer table.
When to use Azure Data Explorer or/and Azure Stream Analytics
Azure Stream Analytics:
- Stream Processing Engine - Continuous/ Streaming real-time analytics
- Job based
- ASA has a lookback window period of 1ms to 7 days for in-memory temporal analytics/stream processing
- Ingest from Event Hub, IoTHub with sub-second latency
Azure Data Explorer:
- Analytical Engine - On-demand/ Interactive real-time analytics
- Streaming + Persistent Store + Query Engine
- Ingest from Event Hub, IoT Hub, Blob, Data Lake, Kafka, Logstash, Spark, ADF, …
- 10sec-5 min latency – recommended for high throughput workload
- Simple data transformation can be done with update policy during ingestion
With this release, you can significantly grow the scope of real time analytics by leveraging ASA and ADX together. Few scenarios:
- Stream Analytics identified anomalies in real time and Data Explorer helps determine how/why it occurred through interactive exploration
- Stream Analytics deserialized incoming data stream for use in Data Explorer (E.g. ingest Protobuff format by using custom deserializer, custom binaries formats etc.)
- Stream Analytics aggregates/filters/enrich/transform incoming data streams for use in Data Explorer
Read more on this announcement on Azure Stream Analytics forum.
Posted at https://sl.advdat.com/3k3psog