Monday, December 6, 2021

Experiencing Data Latency West US region for Metrics - 12/07 - Resolved

Final Update: Tuesday, 07 December 2021 02:03 UTC

We've confirmed that all systems are back to normal with no customer impact as of 12/7, 02:00 UTC. Our logs show the incident started on 12/7, 00:01 UTC and that during the two hours that it took to resolve the issue 20% of customers experienced latency in querying metrics for app services and azure functions, resulting in empty metrics charts and graphs in the Azure Portal.
  • Root Cause: A recent tenant upgrade caused a surge of retries which overloaded a back-end system. This was mitigated by temporarily throttling the query retries, which allowed the backend component to recover.
  • Incident Timeline: 2 Hours - 12/7, 00:01 UTC through 12/7, 02:00 UTC
We understand that customers rely on Azure Monitor Metrics as a critical service and apologize for any impact this incident caused.

-Jack Cantwell

Posted at