Today we announced the general availability of the Azure Monitor Agent (AMA) and Data Collection Rules (DCR) that are two brand new features from the Azure Monitor product. You can read more about the specific changes announced on this launch in this update announcement here.
Both these features address the data collection aspects of Azure Monitor, indicated by the red dotted box below. They aim to provide powerful new capabilities, while at the same time ensuring data collection setup is a breeze for users! Read on to find out how…
What’s the AMA? Is this “yet another agent”?
The AMA collects monitoring data from the guest operating system of virtual machines and delivers it to Azure Monitor. It is meant to replace all other agents that exist today for a similar purpose, consolidating their features and providing more capabilities on top and enabling long-requested asks by all of you.
So, while it is indeed another agent, it’s primary targeted to address the underlying sentiment of this question, which is pain and frustration at “too many agents”. For users today, there’s one agent if you wish to collect telemetry logs data, another agent if you want to collect metrics from Windows machines for alerting, another one if you want to do the same but from Linux machines. I think we all can agree that’s no fun, and the solution to this is (yep, you guessed it), the AMA!
Is the AMA the ‘one agent’ for everything?
The answer is no, but it’s meant to be the single agent for uploading data to Azure Monitor going forward, which collects telemetry data and sends it to Azure Monitor Logs or Metrics (today), and Event Hubs, Storage Accounts and many other destinations that you need to send telemetry data to (in future).
Other extensions need to be deployed for specific “curated” monitoring scenarios like security or insights, that all use the AMA to pump data to Azure Monitor.
Thus, it’s important to understand the new extension model that the AMA introduces that reduce management overhead and provide a consistent, easy way to enable, update, debug and remove specific monitoring scenarios.
- The AMA is available as a VM extension, and so all VM extension capabilities are available out of box like auto-upgrade, no-config installation and more
- Previously known as ‘Log Analytics Solutions’ or ‘Management Packs’, additional VM extensions may also be installed on the machine for specific monitoring scenarios like security monitoring (Azure Security Centre), SIEM (Sentinel), Virtua Machine Insights, Container Insights, VM Guest Health, Network Monitoring and many more
- These extensions all use the AMA extension to pipe data to the required destination, so user can choose to selectively install only what’s required. Nothing more, nothing less!
Okay, what else is new besides consolidating existing feature sets?
- Centralized yet granular ("resource-centric") configuration, that allows you to have a setup that works best for your organization using the power of Data Collection Rules (see next section). You can either opt for centralized collection setup (one rule for all machines) as well as granular collection setup (different rule for different set of machines, even though connected to the same workspace)
- Native to ARM, which means you can use ARM templates, Azure policies, etc to manage monitoring code changes in the same as as your service code, with proper review gates and safe deployment practices
- Event filtering for Windows Event Logs that allows you to limit data collection to exactly what you require, thus providing tremendous cost saving opportunity
"We are very excited with the new Windows Security filtering capabilities in Azure Monitor Data Collection Rules. During the last 4-5 months, 3 of our customers have used the new feature. They have all gained significant logging cost savings (30-50%) being able to exclude irrelevant security events into their log workspace. The XPath filtering capabilities adds great filtering capabilities on agent level, so we only send the needed data into Azure Log Analytics. Moving forward we are also excited to learn, that Microsoft will continue to enhance the filtering capabilities to support some of the features in newer versions of XPath"
- Morten Waltorp Knudsen, CEO & Cloud Architect, 2LINKIT
- Support for multiple destinations, where you collect the data once and send it to multiple Log Analytics workspaces (multi-homing anyone?), or a combination of logs and metrics destinations at the same time
- Much better performance in terms of the EPS (events per second) rate, compared to existing agents
- Seamless experience on Azure and non Azure environments, using Azure ARC as a pre-requisite. The ARC agent comes at no extra cost and is only used to install the AMA and additional VM extensions on your non-Azure machines, the same way it’s done for Azure machines (Hint: One way to manage it all).
“We have a large investment for both on-premise and cloud resources. With Azure ARC and Azure Monitor Agent, we are able to collect real-time data from both environments into log analytics to provide analysis, gain visibility from operation management and compliances. This also allows us to reduce redundant tools, processes, and expenses”
– An avid user from a large US telecommunication company
What’s DCR? Why should I bother learning about it?
“I want an easy way to setup monitoring and data collection for all my resources”
“I want to centrally manage what telemetry data I collect, and just do it one way for all instead of managing it differently per machine or resource type”
“I want a single pane of glass view for any/all data being collected from my resources and how/where it’s being sent to”
“I want a simpler monitoring life! :)”
If you can relate to any of the above, then Data Collection Rules is here to help!
Simply put, a DCR is a rule you can use to define what data to collect, how it should be processed, and where it should be sent to. It’s agnostic of the source of the data as well as the destination, making it independent of either and thus flexible enough to configure collection just the way you need to.
As such, DCR presents tremendous opportunity to revisit and simplify your existing data collection setup, and provides freedom from all existing limitations around configuration to make the most out of Azure Monitor!
How do DCRs work with AMA?
- You first create a DCR which is nothing but an Azure ring 1 resource, which resides within Azure in a resource group and subscription of your choice (same as any other resource)
- In the DCR, you select which types of data you’d like to collect from what’s supported today
- Finally you specify the destination(s) where the data needs to be sent to. Within a single DCR, you can specify multiple Log Analytics workspaces today or a combination or logs and metrics destinations in Azure Monitor. Note: The metrics destination is not GA and continues to remain in preview with these limitations
- Once created, one or more DCRs can be applied to one or more machines using DCR-Association (or DCRA) which need to be deployed per machine (1:1 mapping between DCRA and resource)
- These machines should have the AMA installed which then polls into new Azure Monitor Control Service to fetch the DCRs the machine is associated with. Once received, AMA starts collecting the required data
- AMA uses Managed Identity so the machines should have that enables. See prerequisites here.
Note: When using the DCR creation experience on Azure Portal (shown below), you only need to perform step #1. All other steps are performed on your behalf (woohoo, no need to worry about agent installation!)
Does it only support AMA?
As of today, DCR is used to configure what the AMA should collect and where the data should be sent to. In the long term though, Data Collection Rules will become the single, consistent way to configure data collection for other resources like PaaS services (uses Diagnostic Settings today) or application (uses AppInsights SDK today), and whatever else that you need. Tell us now and help us build it for you!
What’s the migration story from existing agents?
We will not be auto migrating customers due to the fact that AMA is not at full parity yet with the existing agents.
- Log Analytics Agents customers may start migrating to AMA if their required set of capabilities and “solutions” are available already. Migration can be performed using the same steps listed in our documentation today, and using policies to replicate these steps "at scale"
- In addition, ensure to cleanup existing agents, workspace keys, config files from machines
- Also disable solutions on workspaces that you wish to use the new agent for instead
- Azure Diagnostic Extension and Telegraf agent customers can start migrating to AMA if they only need to send metrics data to Azure Monitor Metrics. Sending data to Event Hubs and Storage accounts is not supported yet.
We will be releasing migration tools in the coming months to automate and simplify some of this for you. In the meantime, if you need additional guidance or have questions, please reach out to the product group directly. We would be happy to assist!
What’s next?
This launch is just the first step towards a better and more powerful world of data collection for monitoring.
On the agent side:
- We continue to consolidate features from existing agents, including but not limited to:
- Custom log collection
- Windows 10 support
- New destinations like event hubs and storage accounts
- Support for popular solutions (AMA extensions) that you use today. This should unblock more of you to migrate soon to the better agent.
On the DCR side, we will expand scope to act as the control plane for other parts of Azure Monitor data collection, like API based ingestion and Diagnostic Settings.
