Web activity is helpful when invoking an external endpoint from within an ADF pipeline. While Azure Data Factory/ Synapse pipelines offer various orchestration activities, web activity helps provide extensibility when building data pipelines that span across different services.
Web activities work exceptionally well with APIs that follow the asynchronous request-reply pattern, a suggested approach for building scalable web APIs/ services. In such scenarios, it automatically polls for the status endpoint (when it receives HTTP 202) till the API returns HTTP 200. It behaves synchronously and waits till the response is HTTP 200 before kicking off the down-stream activities.
Most Azure-based APIs are async request-reply pattern and gels with the existing web activity behavior.
Problem
But in case you have API endpoints that do not follow the above pattern and expect the caller to wait for a response till the processing/ actions are done. As default action, the web activity will timeout within 1 minute if it does not receive any response.
Improvement
We are introducing a response timeout property 'httpRequestTimeout' in Web activity that will let you provide larger timeouts for such synchronous APIs.
Reference:
Web Activity - Azure Data Factory & Azure Synapse | Microsoft Docs
Posted at https://sl.advdat.com/3icwDcthttps://sl.advdat.com/3icwDct