Final Update: Tuesday, 07 December 2021 02:03 UTC
We've confirmed that all systems are back to normal with no customer impact as of 12/7, 02:00 UTC. Our logs show the incident started on 12/7, 00:01 UTC and that during the two hours that it took to resolve the issue 20% of customers experienced latency in querying metrics for app services and azure functions, resulting in empty metrics charts and graphs in the Azure Portal.
We've confirmed that all systems are back to normal with no customer impact as of 12/7, 02:00 UTC. Our logs show the incident started on 12/7, 00:01 UTC and that during the two hours that it took to resolve the issue 20% of customers experienced latency in querying metrics for app services and azure functions, resulting in empty metrics charts and graphs in the Azure Portal.
- Root Cause: A recent tenant upgrade caused a surge of retries which overloaded a back-end system. This was mitigated by temporarily throttling the query retries, which allowed the backend component to recover.
- Incident Timeline: 2 Hours - 12/7, 00:01 UTC through 12/7, 02:00 UTC
-Jack Cantwell
