Here is a situation. Let's say we cache the last month of data in Azure Data Explorer (ADX). This is great for our operational queries which are all done within that last month of telemetry: it limits the on-cluster resource we consume while making sure our operational queries are snappy fast. Now an audit comes in requiring us to look at the entire year of 2019 (2 years ago).
What can we do?
The fundamental caching image for a Kusto table is as follow (assuming we have 5 years of accumulated data):
Data is persisted in cold storage and table's cache policy dictates how much of the most recent data gets cached in hot cache on the cluster nodes. The hot cache offers the best query performance: an order of magnitude more performant than cold data. Cache policy can be changed at the database or table level and quickly change the amount of data loaded in hot cache. This allows cluster operator to easily control how to allocate cluster resources for different workloads.
In our scenario we want to query that cold data. This is possible without any modification. For queries done on a narrow time window, for instance aggregating events of one day of data, the performance are quite reasonable. Those are called point in time queries or point queries and can be run at any time.
If we want to run queries on large amount of data in the past, those will not be point queries. Let's say in our scenario we would need to run aggregations or analysis (e.g. anomaly detection) over the entire year of 2019. Those would not perform well against cold data.
We could update the cache policy to go back three years ago. This would include the year 2019 and enable fast queries over the data of that year. That would also include the year 2020 and 2021 we do not need though and it might consume more hot cache than we have available on our cluster at the current scale.
A new feature we recently introduced is quite useful here: hot windows. A hot window is a time window in the past we want to be part of the hot cache. It doesn't need to be contiguous with the "main hot cache" (latest data). There can be multiple hot windows for the same table.
A hot window is defined within the cache policy.
To resolve our situation, we would simply define a hot window for the year 2019. Once we did our inquiry, we could remove that hot window.
This feature is well suited for forensic exploration where we need to double click into a certain time window in the past and have a team of analyst be productive quickly.
We just published an article in the Azure Data Explorer documentation about hot windows.
Posted at https://bit.ly/3bpYNgt
