Advanced Data Solutions : Ingestion Cost Spike detection App

Azure Sentinel is a modern SIEM solution offering cloud scale analytics to power your thereat detection and response requirements. Like most cloud solutions , the billing for Azure Sentinel is largely based on a pay per use model. Specifically for Azure Sentinel, billing is based on the amount of data ingested into Log Analytics and Azure Sentinel. To ensure that you have continuous visibility should the amount of billable data ingested into the platform experience an unexpected spike, we have developed this Logic App to address exactly this sort of scenario.

This ingestion cost spike alert logic app is based on the principle of anomaly detection and as such utilizes the built-in KQL function series_decompose_anomalies(). It compares the base line/expected level of ingestion over a period of time and then uses that historical pattern to determine whether to alert on a sudden increase of billable data into the workspace. Below is an image depicting the various actions the Logic App steps through, followed by a detailed explanation of the key parts of the query that checks for anomalies based on the historical ingestion pattern . The Logic app is triggered on a recurring schedule. Since you probably want to be immediately notified when this type of anomaly occurs then you may want to run it on a daily basis.

image of logic App overview

let UpperThreshold = UpperAnomalyThreshold; //+3 is the suggested number and it indicates a strong anomaly though you can modify it : Outlier - Wikipedia

Usage
| where IsBillable == "true" //we are only interested in tables getting notified when a spike is detected in a billable table 
| where Quantity > ReportingQty //Allows you to report only on variations that are above a certain threshold that you deem significant enough to warrant an alert
| make-series Qty=sum(Quantity) on TimeGenerated from ago((LookBack)d) to now() step 1d by DataType //creates a time series to look at the ingestion pattern over the period defined in the LookBack variable
| extend (anomalies, score, baseline) = series_decompose_anomalies(Qty, 1.5, 7, 'linefit', 1, 'ctukey', 0.01) //takes the time series of ingested data across the days specified in the ‘LookBack’ variable and extract anomalous points with scores based on predicted values using the linear regression concept. See https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/series-decompose-anomaliesfunction for a detailed explanation of each argument. For an explanation of 'ctukey' read: Outlier - Wikipedia.

| where anomalies[-1] == 1 or anomalies[-1] == -1 //the output of series_decompose_anomalies function is three things: A ternary (as opposed to binary) series containing (+1, -1, 0) marking up/down/no anomaly respectively, the Anomaly score and the predicted value or baseline. 

| extend Score = score[-1] ] //this part picks up the anomaly state from the most recent run. -1 indicates a position in the array. 

| where Score >= UpperAnomalyThreshold  //compare with strong anomaly indicator values extracted from the time series data

| extend PercentageQtyIncrease = ((round(todouble(Qty[-1]),0)-round(todouble(baseline[-1]),1))/round(todouble(Qty[-1]),0) * 100) //calculates percentage increase to present data in percent terms for easier appreciation of the anomaly
| project DataType,ExpectedQty=round(todouble(baseline[-1]),0), ActualQty=round(todouble(Qty[-1]),0),round(PercentageQtyIncrease,0)  
| order by  round(todouble(PercentageQtyIncrease),0) desc 
| where PercentageQtyIncrease > PercentIncrease //only alert if the percentage increase exceeds the threshold beyond which you specified that you wish to be notified

Note: This logic app is complementary to the previously released Ingestion Cost Alert App but different in function. The Ingestion Cost Alert App is designed to send you alerts if the budget you define is exceeded. In contrast, the Ingestion Cost Anomaly App is designed to alert you, should there be an unusual spike in the billable data being ingested into the Log Analytics workspace where you have deployed Azure Sentinel. The App provides you with the flexibility to determine two thresholds around which the alerting should occur:

The minimum increase in the amount of data in Giga bytes around which alerting should occur. This allows you to suppress alerts triggered by increases you consider immaterial
The percentage increase in data. This parameter gives you additonal flexibility to manage alerting thresholds by specifying what percentage increase you consider worth triggering the anomaly alert on.

To deploy the Ingestion Cost Anomaly App, follow this link to our GitHub repo. As part of the deployment process, you will need to specify some parameters in the “project details” page that determine the sensitivity of the App in terms of how it responds to ingestion anomalies, as well as define additional settings specific to your environment. See below highlighting the various parameters needed in this form:

Upon a successful run of the logic app and should there be a billable data ingetsion spike in your workspace then an e-mail with contents similar to the below will be sent out to the designated recipients :