Introduction:
In today Blog article, we will go through technical steps to connect Azure Data factory to SQL Server on Azure virtual machine across different subscription/VNets.
Use Case:
Customer wants to connect Azure Data Factory on one subscription to an Azure SQL Server on Virtual Machine (SQL VM) on another subscription. check out the architecture diagram below for more clarification.
Solution:
Here, I am listing the main steps of this set up:
-- VNET Peering
-- Data factory Private EP
-- Private zone DNS resolution
-- Self-Hosted Integration Runtime
- VNet Peering
|
Set up the VNet peering between the two VNets using the steps in article: Connect virtual networks with VNet peering - tutorial - Azure portal | Microsoft Docs
More information can be found at: |
- Data factory Private Endpoint
|
After change the network access to private endpoint, Set up a private endpoint link for Azure Data Factory using the steps: Azure Private Link for Azure Data Factory - Azure Data Factory | Microsoft Docs |
- DNS resolution
|
To connect privately with your private endpoint, you need a DNS record. We recommend that you integrate your private endpoint with a private DNS zone. You can also utilize your own DNS servers or create DNS records using the host files on your virtual machines.
You can link a private DNS zone to one or more virtual networks by creating virtual network links. You can also enable the autoregistration feature to automatically manage the life cycle of the DNS records for the virtual machines that get deployed in a virtual network. |
|
More information can be found at: |
Note:- If you are using a custom DNS server on your network, clients must be able to resolve the FQDN for the Data Factory endpoint to the private endpoint IP address. You should configure your DNS server to delegate your private link subdomain to the private DNS zone for the VNet, or configure the A records for ' DataFactoryA.{region}.datafactory.azure.net' with the private endpoint IP address. For more information on configuring your own DNS server to support private endpoints, refer to the following articles:
Name resolution for resources in Azure virtual networks | Microsoft Docs
What is a private endpoint? | Microsoft Docs
- Self-Hosted Integration Runtime
Download and install Self-hosted Integration runtime on Azure VM. Download Microsoft Integration Runtime from Official Microsoft Download Center
Configure and register the SHIR with Azure data factory. Create a self-hosted integration runtime - Azure Data Factory & Azure Synapse | Microsoft Docs |
|
Make sure the Self-hosted node is connected to the cloud service. |
Troubleshooting:
Issue #1:
Self-hosted failed with error like below: The access to data factory from public network is blocked
Possible Solution:
Add the private endpoint to host file or make sure the DNS resolution.
When you set up your ADF to connect to the VNet using a Private Endpoint there are DNS changes that occur. If you are using a custom DNS server on your network, clients must be able to resolve the FQDN for the Data Factory endpoint to the private endpoint IP address. You should configure your DNS server to delegate your private link subdomain to the private DNS zone for the VNet, or configure the A records for ' DataFactoryA.{region}.datafactory.azure.net' with the private endpoint IP address. More information at:
Tip:
you might need to perform flush dns cache after updating DNS or the Host file under system32/drivers/etc/host using command:
ipconfig /flushdns
Additional References:
Azure Private Link for Azure Data Factory - Azure Data Factory | Microsoft Docs
Thanks to @abhishekshaha and @Marcio Lipi for helping to create this content.
I hope you find this article helpful. If you have any feedback, please do not hesitate to provide it in the comment section below.
Ahmed S. Mazrouh
Posted at https://sl.advdat.com/3IT03rDhttps://sl.advdat.com/3IT03rD