This installment is part of a broader learning series to help you become a Jupyter Notebook ninja in Azure Sentinel. The installments will be bite-sized to enable you to easily digest the new content.
- Part 1: What are notebooks and when do you need them?
- Part 2: How to get started with notebooks and tour the features
- Part 3: Overview of the pre-built notebooks: the Grand List – this post
- Part 4: How to create your own notebooks from scratch and how to customize the existing ones
Through Part 1 and Part 2 of this Azure Sentinel Notebook Ninja series, we’ve discussed the concepts and activities to best become acclimated with Jupyter notebooks for Azure Sentinel. The next step in our process is understanding the value of having ready-made notebooks ready for use as part of the solution.
When a customer stands-up Azure Sentinel for the first time, there are a number of additional pieces of ready-to-use collateral that are provided out-of-the-box including Analytics Rules, Hunting queries, Connectors, Solutions, Workbooks – and – you guessed it – Notebooks.
The notebooks are mostly one of three types:
- Exploration notebooks. These are meant to be used as they are or with your own customizations to explore specific hunting and investigation scenarios. Examples of this type include the Entity explorer series. (“Entity” refers to items such as hosts, IP addresses, accounts, URLs, etc.)
- Simple How-To notebooks like the Get Started notebook.
- Sample notebooks. These are longer and are meant to be instructional examples following a real or simulated hunt or investigation.
The following Grand List represents the notebooks that are provided when setting up Azure Sentinel.
Azure Sentinel Notebooks – The Grand List
|
Notebook Name |
Type |
Description |
Special Requirements |
|
|
|
|
|
|
A Getting Started Guide For Azure Sentinel ML Notebooks |
Getting Started |
This notebook guides you through the basic steps of using notebooks for security analysis. It covers all the basic steps you need to understand to start using the notebooks provided with Azure Sentinel. |
Data type(s): SigninLogs
External service(s): VirusTotal, MaxMind, MSTICPy |
|
A Getting Started Guide For PowerShell AML Notebooks |
Getting Started, PowerShell |
This notebook takes you through the basics needed to get started with PowerShell notebooks that leverage Azure Sentinel data and APIs. |
Data type(s): SecurityEvent
|
|
A Tour of Cybersec notebook features |
Getting Started |
This notebook takes you through some of the features of Azure Sentinel notebooks and MSTICPy. It's a good second notebook to explore after "A Getting Started Guid for Azure ML notebooks". It covers data queries, visualization, data analysis, enrichment with threat intelligence and pivot functions. It can be run against you Azure Sentinel workspace or standalone, with sample data. |
Data type(s): SecurityEvent, SecurityAlert
If no data types are available, sample data will be used.
External service(s): MSTICPy |
|
Configuring your Notebook Environment |
Configuration |
This notebook takes you through detailed setup of your settings for Azure Sentinel Notebooks and the MSTICPy library. It covers: Setting up your Python environment for notebooks (not required for AML notebooks), Creating and editing your msticpyconfig.yaml file, Understanding and managing you config.json file. |
External service(s): MSTICPy |
|
Credential Scan on Azure Blob Storage |
Hunting |
This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Blob Storage using Azure SDK for Python. |
External service(s): MSTICPy |
|
Credential Scan on Azure Data Explorer |
Hunting |
This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Data Explorer using Azure SDK for Python and KQL. |
External service(s): MSTICPy |
|
Credential Scan on Azure Log Analytics |
Hunting |
This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Log Analytics using Azure SDK for Python and KQL. |
Data type(s): SecurityEvent
External service(s): MSTICPy |
|
Entity Explorer – Account |
Hunting |
Use queries and visualizations to help you assess the security state of an AAD/O365 account or an account on a local host. It includes examining Azure Active Directory (AAD) and Office365 activity for an account and identifying any related anomalous behavior. Allows you to correlate the IP addresses in related events with threat intelligence sources. |
Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, AAD, OfficeActivity
External service(s):kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2 |
|
Entity Explorer - Domain and URL |
Hunting |
Use queries and visualizations to help you assess the security state of a DNS host or domain name. Examine DNS requests to look for malicious DNS usage, such as DNS tunneling, as well as anomalous DNS name lookups. Query threat intelligence and domain ownership using WhoIS information for a specific DNS name, to get a better understanding of the domain name’s reputation. |
Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, DNS, Syslog
External service(s): kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2 |
|
Entity Explorer - IP Address |
Hunting |
Brings together a series of queries and visualizations to help you assess the security state of an IP address. It works with both internal addresses and public addresses. |
Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat
External service(s): VirusTotal, Alienvault OTX, IBM Xforce |
|
Entity Explorer - Linux Host |
Hunting |
This notebook brings together a series of tools and techniques to enable threat hunting within the context of a singular Linux host. The notebook utilizes a range of data sources to achieve this but in order to support the widest possible range of scenarios this Notebook prioritizes using common Syslog data. |
Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat
External service(s): kqlmagic, msticpy, pandas, pandas_bokeh, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2 |
|
Entity Explorer - Windows Host |
Hunting |
Brings together a series of queries and visualizations to help you determine the security state of the Windows host or virtual machine that you are investigating. It looks for related alerts, allows you to examine logon sessions and processes, check process command lines for IoCs and explores network traffic between the host and external endpoints. |
Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat
External service(s): kqlmagic, msticpy, pandas, numpy, matplotlib, bokeh, networkx, ipywidgets, ipython, scikit_learn, dnspython, ipwhois, folium, maxminddb_geolite2 |
|
Guided Hunting - Anomalous Office365 Exchange Sessions |
Hunting |
Brings together a series of data science techniques to help you hunt for anomalous sessions in your Office Exchange logs. It queries the OfficeActivity table, creates sessions from the PowerShell cmdlets (e.g. Set-Mailbox), trains a model and then visualizes the sessions. |
Data type(s): OfficeActivity
External service(s): msticpy, pandas, kqlmagic |
|
Guided Hunting - Base64-Encoded Linux Commands |
Hunting |
This notebook is a collection of tools for detecting malicious behavior when commands are Base64-encoded. It allows you to specify a workspace and time frame and will score and rank Base64 commands within those bounds. It utilizes multiple data sources, primarily focusing on Azure Sentinel Syslog data augmented by telemetry from the MSTIC research branch of the AUOMS audit collection tool. |
Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL
External service(s): kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, folium, maxminddb_geolite2, BeautifulSoup |
|
Guided Hunting – Covid-19 Themed Threats |
Hunting |
Brings together a number of techniques and approaches to allow an analyst to hunt for indicators of Covid-19 themed threats within an organization's data. This includes hunting templates for network, cloud and host logs in order to find potentially malicious Covid-19 themed activity. |
Data type(s): SecurityEvent, OfficeActivity, CommonSecurityLog
External service(s): MSTICPy |
|
Guided Investigation - Anomaly Lookup |
Investigation |
Gain insights into the possible root cause of an alert by searching for related anomalies on the corresponding entities around the alert’s time. This notebook will provide valuable leads for an alert’s investigation, listing all suspicious increase in event counts or their properties around the time of the alert, and linking to the corresponding raw records in Log Analytics for the investigator to focus on and interpret. |
Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Auditd_CL, OfficeActivity, CommonSecurityLog
External service(s): MSTICPy |
|
Guided Investigation - Process Alerts |
Investigation |
This notebook is intended for triage and investigation of security alerts. It is specifically targeted at alerts triggered by suspicious process activity on Windows hosts. Some of the sections will work on other types of alerts but this is not guaranteed. |
Data type(s): SecurityEvent
External service(s): OTX, VirusTotal, XForce |
|
Guided Investigation - Solarwinds Post |
Investigation |
This notebook assists defenders in hunting for SolarWinds post compromise Tactics , Tools and Procedures (TTPs) across different environments both on-premises and cloud data sources. |
Data type(s): SecurityEvent, AuditLogs, SigninLogs, OfficeActivity
External service(s): OTX, VirusTotal, XForce |
|
Guided Investigation – Alert Triage |
Investigation |
Rapidly triage alerts raised by a range of sources (Azure Sentinel, MDE, MCAS, ASC, etc.) by enriching alert data with Threat Intelligence and OSINT. This notebook is intended to be used by analysts as part of a standard operating procedure. The notebook uses UI widgets and simple workflow to ensure an easy introduction to notebooks for all levels of analyst levels. This Notebook uses MSTICpy to connect to Threat Intelligence sources |
Data type(s): SecurityAlert
External service(s): MSTICPy |
|
Guided Web Shell Investigation - MDE Sentinel Enrichments |
Investigation |
This notebook investigates Microsoft Defender for Endpoint (MDE) web shell alerts. The notebook will guide you through steps to collect MDE alerts for web shell activity and link them to server access logs to identify potential attackers. |
Data type(s): SecurityAlert, W3CIISLog
External service(s): MSTICPy |
|
Hands on - Data Discovery using Azure REST API |
Getting Started |
This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel data discovery by using Azure REST API. |
Data type(s): SecurityEvent |
|
Hands on - Surfing Your Data using Azure SDK for Python |
Getting Started |
This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel log data discovery by using Azure SDK for Python and Kusto Query Language (KQL). |
Data type(s): SecurityEvent |
|
Machine Learning in Notebooks Examples |
Getting Started |
This notebook template guides you through using time series analysis to detect anomalous network activity, clustering to highlight unusual logon sessions, and using Markov Chain to identify anomalous sequences in events. |
Data type(s): SecurityEvent
External service(s): MSTICPy |
These notebooks and more also exist in the official Azure Sentinel GitHub repository. This repository contains notebooks contributed by Microsoft and the community to assist hunting and investigation tasks in Azure Sentinel. There are a number of notebooks in the GitHub repository that are valuable additions, but are not supplied automatically when you stand-up Azure Sentinel. You should regularly review this repository for more, or use the RSS feed to be notified of additions and changes.
Notebooks, like all other components and features for Azure Sentinel, are under constant review and undergoing constant improvement. Improvements and changes come from feedback and suggestions from our customers. You can use the DL (asinotebooks@service.microsoft.com) to send your questions, issues, and feedback and our various product teams will monitor and respond.
As these updates are made available, we’ll update the Grand List.
We are super-excited to be bringing this series (and the training) to you! Look for more great knowledge on Azure Sentinel Notebooks as we prepare to deliver Part 4 of this series: How to create your own notebooks from scratch and how to customize the existing ones.
More reading/tutorial resources:
- Notebooks at the Azure Sentinel GitHub Repository
- Use Jupyter Notebook to hunt for security threats
- Detect Credential Leaks using Azure Sentinel Notebooks! (video)
- Azure Sentinel: Notebooks - Getting Started (video)
- Detect Malicious Base64-Encoded Commands on Linux Hosts (video)
- Azure Sentinel Weekly Newsletter
Posted at https://sl.advdat.com/3lPcavL
