Thursday, March 17, 2022

Web activity response timeout improvement

Web activity is helpful when invoking an external endpoint from within an ADF pipeline. While Azure Data Factory/ Synapse pipelines offer various orchestration activities, web activity helps provide extensibility when building data pipelines that span across different services. 


Web activities work exceptionally well with APIs that follow the asynchronous request-reply pattern, a suggested approach for building scalable web APIs/ services. In such scenarios, it automatically polls for the status endpoint (when it receives HTTP 202) till the API returns HTTP 200. It behaves synchronously and waits till the response is HTTP 200 before kicking off the down-stream activities. 



Most Azure-based APIs are async request-reply pattern and gels with the existing web activity behavior. 



But in case you have API endpoints that do not follow the above pattern and expect the caller to wait for a response till the processing/ actions are done. As default action, the web activity will timeout within 1 minute if it does not receive any response.  



We are introducing a response timeout property 'httpRequestTimeout' in Web activity that will let you provide larger timeouts for such synchronous APIs.  


Screenshot 2022-03-17 at 9.33.41 AM.png



Web Activity - Azure Data Factory & Azure Synapse | Microsoft Docs

Posted at