Auto-delete on idle: Service Bus (SB) Queues or Topic-Subscriptions are automatically deleted & the SB client application fails with error code 404 (Not Found) or MessagingEntityNotFoundException.
Scenario:
The applications using Service Bus (SB) Queues or Topic- Subscriptions are failing with error code 404 (Not Found) or MessagingEntityNotFoundException . On investigation, you find that SB Queues or Topic- Subscriptions are suddenly removed or automatically deleted after it has been working for a while.
You have SB queues or topics-subscriptions that were provisioned. It was working without problems until now. You encounter errors\exceptions that indicate that the entity “does not exist” or is “not found”. You investigate and find that the entity is missing. You check with your colleagues, your Azure administrator if they have deleted the SB queues or subscriptions and they reply negatively.
You start wondering…Why did the queues or subscriptions suddenly go missing?... Why are they suddenly removed or automatically deleted?
Cause:
One of the properties that can be set on SB Queues or Topic-Subscriptions is Auto-delete on idle. Autodelete on idle enables you to specify an idle interval after which a queue or topic subscription is automatically deleted.
SB Queue, Topic-Subscription can use Auto-delete on idle property to make itself Temporary i.e., which will be removed if they are not used for a specified period. You can read more about Temporary Entities here. You will find what is considered Idleness here for Queues, Topics-Subscriptions.
Verification:
- During the creation of the Entity (Queue, Topic-Subscription) from the portal you would have Checked - Enable auto-delete on idle queue and selected the Time-Span. After provisioning the entity, you will see Auto-delete after idle in the Overview page of the Entity. However, if the entity is already deleted, as may be in your case, this view may not be available for you to cross-verify.
- In such a case where the entity is already deleted, but if you have saved the ARM Template for the SB Resource you can fall back on it verify the autoDeleteOnIdle value, and recreate the entity. (You are expected to save the ARM Template in your Source Control, at least for your Production Environment Azure Resources).
- In case you do not have the above-mentioned artifacts (#1 & #2), then we will have to rely on conjecture. You can navigate to the Overview page for the entity, and in the Metrics, section select Incoming Messages and Outgoing Messages. If there are no incoming or outgoing messages for a considerable amount of time, then you may surmise that Auto-delete on idle may have set in and deleted the entity. (Considerable amount of time is subjective.)
Questions:
- So, is there no way SB users can know for sure if Auto-delete on idle has kicked in?
- Of course, there are ways to identify if the Auto-delete on idle has transacted… There is a proactive approach (being ready to record events before they happened) and a reactive approach (trying to check things after the fact).
The Proactive Approach:
The Proactive Approach is to enable Diagnostic Settings at the SB Namespace Level, esp. the OperationalLogs. These logs can be viewed from Log Analytics using simple queries. You can look for Auto Delete events to check if the Auto-delete on idle has set in.
The Reactive Approach:
The SB users can always open a Support Ticket with Microsoft from Azure Portal. Kindly provide the below details while opening a case for such an issue.
Required Information:
- What is the Subscription Id, Service Bus Namespace Name, Name of the affected Entity i.e., Queue or Topic-Subscription?
- When was the queue or topic-subscription created (optional)? When was the last time you had observed that the entity was working fine?
- Exact date and time when you observed that the queue or topic-subscription went missing?
- What prompted you to check the queue or topic-subscription? (Did your clients complain about any errors\exceptions\functional misbehavior, that made you relook at the entities? – Please specify)
- Is Auto-delete on idle the only reason why Queues or Topic-Subscriptions may be suddenly removed?
No, there could be various other reasons why Queues or Topic-Subscriptions may be suddenly removed… Let me state them below from simple to complex scenarios (This is not an exhaustive list):
Reasons why Entity may suddenly go missing
How to prevent it?
The user may delete the entity in error and not inform the team, via Portal, CLI, PowerShell, or RESTAPIs. Be careful to whom you provide write\delete privilege. Follow the principle of least privilege. Only users with write privilege will be able to see/access the SAS keys and those with delete privilege can delete resources.
You can check in the ARM Activity Logs if a user has deleted the entity via Portal, PowerShell, or CLI.
Use ARM Resource Lock to prevent undesirable changes to SB Entities.
Applications can make use of SB Resource Provider (RP) APIs to delete the Entities.
( SB Resource Provider APIs calls are not logged in Azure Logs like Activity, Operation, Diagnostic, etc. It is the responsibility of the applications making SB RP API calls to log the operations it is performing.)
Service Bus libraries & tools like ServiceBusExplorer use these SB RP APIs.
Refer: Manage using Azure Resource Manager-based libraries & Manage using Service Bus client libraries.
Important Note: ARM Resource Lock placed on SB Namespace\Entities are not protected when performing CRUD operations via SB SDK. This is because the SB SDK directly talks to SB’s Resource Provider and does not route via ARM.Instead of using SAS Keys in your application, consider using Role-based Authentication and Access. You can Disable local or shared access key authentication with Azure Service Bus.
Refer:
Managed identities for Azure resources with Service Bus
Authenticate an application to access Azure Service Bus entitiesHave proper logging in your application, esp. when CRUD operations are performed. Also, Turn on OperationalLog in Diagnostics settings.
Auto-delete on idle.
Discussed in detail in this article.
SAS Key\Token is Compromised
Follow the principle of Least Privilege.
Keep audit of SAS Key distribution. If SAS Key is compromised immediately recycle the Keys.
Others
Open a Support Ticket with Azure and provide all the relevant details.
- Does Azure Support Team have any insights on which Client Application deleted the entity?
We do not collect application or server information that performs the delete operation. So, the support professionals will not be able to provide information regarding Application\Server that made the delete request.
The customers can configure VNets and firewalls to control the inflow of requests and monitor them. This has helped some customers identify the clients (machines) making these requests. VNetAndIPFilteringLogs and OperationalLogs in the Diagnostic Setting will come in handy. - Is there any scenario when Auto-delete on idle property should not be set?
The obvious answer is if you expect the entity to be perpetual (until it is manually deleted), and that you do not worry about its existence irrespective of incoming and outgoing messages.
Auto-delete on idle is usually used in Dev-Test environment, or when entities are dynamically created and deleted. It prevents resource leaks, that may have been caused by exceptions\errors in the clean-up code path.
Also, there is one other scenario where Auto-delete on idle should not be set i.e. when the entities are provisioned for GeoDR. This is simply because entities created in the secondary namespace will not receive any messages, thus deleting those entities if auto-delete on idle is set.
