Case:
In some scenarios, we may observe that consumers fail to receive the messages from Azure service bus and the messages are not present within the active queue, even though there is a successful inflow of messages at the producer end into Azure Service bus.
One of the common possibilities for the above behavior is, “Messages could be dead-lettered and thereby sent to dead-letter queue”.
The purpose of the dead-letter queue is to hold messages that can't be delivered to any receiver, or messages that couldn't be processed. The dead-letter queue doesn't need to be explicitly created and can't be deleted or managed independent of the main entity.
We can find out the reason for dead lettering using the service bus explorer tool. A preview version of this tool is available at Azure portal itself.
Steps to verify the dead-letter reason at the portal:
- Go to Service bus namespace -> Go to the affected Service bus Queue -> Click on Service bus explorer tool -> Under peek option, select dead letter -> you will see many dead lettered messages.
- Click on any one of the dead lettered messages and under custom properties you can see the reason behind the dead lettering.
- When you click on the message you will see custom properties like this along with 'name' and 'value' pair. You can get the reason over there for the dead lettering.
Reasons for dead lettering to happen:
- Refer the link to go through the different reasons and how to mitigate them. Service Bus dead-letter queues - Azure Service Bus | Microsoft Docs
As in most scenarios, dead lettering happens with the reason "Max delivery count exceeded". So, giving a few insights to help if max delivery count is the dead-letter reason.
Solution to overcome Max delivery count Exceeded:
- At the application end, when you are receiving the messages in peek lock receiving mode, you should perform the complete operation within the lock duration. Otherwise, you can enable renew lock operation as well.
- Max delivery count per message will be incremented when they couldn't send the complete request within the lock duration.
- During this, when the Max delivery count limit is crossed, this will lead to dead lettering the message with the reason saying that Max delivery count Exceeded.
Required concepts to follow the above scenario:
What is Peek-lock receiving mode:
- Service bus works in a way that one message per only one receiver.
- While receiving the messages in peek lock mode, a lock will be set on the message, which has been received by the consumer, till the lock duration. (Restricting from other receivers from receiving)
- The consumer must send the complete request/ renew request within the lock duration saying that they have successfully received the message and then it will be deleted from the queue.
- Otherwise, the messages will come out of the lock and will be available for other receivers and Max delivery count for that message will be incremented by 1.
- If Max delivery count is set to 5, that message has 5 retries before moving it to the dead letter queue. (If dead lettering is enabled)
- To learn more on peek lock receiving mode, refer to the document below.
