Azure DevOps is the service of choice that you can use to automate the provisioning of your modern data platform and to allow the continuous integration (CI) and continuous delivery (CD) to continuously test, build , and deploy your code across multiple environments.
In the particular context of Azure Synapse Analytics service, you can think of CICD as a two staged process:
- A first stage, where you deal with resources such as compute pools and the workspace itself
- A second stage, where you deal with the so called artifacts (Synapse Notebooks, SQL Scripts, pipelines, etc)
In my previous article, I demonstrated how we could easily use the Synapse Workspace Deployment extension to accomplish this second stage of the process. I’m now coming back to this topic as I realized that many of our customers were reporting difficulties in completing this second stage of their Synapse CICD process because they were failing to deploy Synapse artifacts to a Managed VNET Synapse Workspace.
In this particular scenario, the deployment was failing because public network access to their target workspace has been disabled.
The deployment was failing with the following error:
Deploy artifact failed: {"code":"PublicNetworkAccessDenied","message":"The public network interface on this Workspace is not accessible. To connect to this Workspace, use the Private Endpoint from inside your virtual network or enable public network access for this workspace."}
To address this specific problem, a solution would be to combine the usage of Private Endpoints and Azure DevOps Self-hosted agents. I’m providing below a two phased action plan with the detailed steps to achieve this :
In a first phase, it will be detailed the necessary configuration process when it comes to configure the DevOps side of the equation
The second phase will focus on the necessary networking configuration.
Stage 1 – Configure DevOps
First things first: you need to use our own compute infrastructure that will run the Azure DevOps Release Pipeline. Remember that your Synapse workspace is blocking access from public networks, so instead of using a Microsoft-hosted that will try to reach the workspace via public endpoint, you will need to use a Self-hosted agent and sit this agent on a VNET so it can communicate with the Workspace on a private and secure way.
Let’s start by creating a Self-hosted Windows agent (you can refer to this public doc to get more details about this step). In this example, you'll need an Azure VM to install this agent:
- Connect to your Azure VM and open a new browser session. Navigate to https://dev.azure.com/{your_organization} and Sign in with the user account you plan to use in your Azure DevOps organization
- If you haven’t created your your personal access token (PAT) please follow the instructions provided here.
- Download and configure the Self-hosted agent following this instructions
This screenshot below reflects an example of this configuration step:
When you have finished the configuration process above, check your “Default” agent pool in DevOps and you should see your newly created Self-hosted agent Online:
If you still see your agent Offline, make sure to run the agent from the VM, submitting the following command from the Powershell:
.\run.cmd
Once you have your Self-hosted agent up and running, it’s time to configure the Release Pipeline.
- Follow these steps below to configure your Release Pipeline to deploy Synapse Workspace artifacts from a source environment to a target environment:
From the left menu select “Pipelines-->Releases” and then select “New release pipeline”
Select to start with an “Empty job” when selecting your template. Provide a new name to your Release Pipeline and also to the initial stage. Click on the “+ Add” to add a new Artifact.
Configure the Pipeline source artifact by providing your Project, Source Repository and Default branch (typically the workspace branch, where you have your workspace JSON files published).
Select “Add” to confirm your settings and to create the source artifact. Now it’s time to configure the Release Stage. Select the “1 job , 0 task” link on your stage.
Provide a new display name to your agent. Once thing important is to change the “Agent selection” from “Azure Pipelines” to “Default” (this is the pool that is hosting your newly created Self-hosted agent)
Click the “+” sign on the Agent job bar to add a new task. Then search for Synapse and select “Add” to add the "Synapse Workspace Deployment" task to your agent.
Configure the Synapse Workspace deployment task (for the sake of this demo I’m not overriding any parameters) and don’t forget to save your changes
At the end of this first stage you should have have finished the following action items:
- The Self-Hosted up and running
- The Release Pipeline configured
Let’s move forward to the final stage of this article: putting all networking pieces together so the Azure DevOps service and Synapse service can communicate privately and securely.
Stage 2 – Network Setup
Navigate to your Synapse Workspace resource page and create a Private Endpoint to the Synapse Workspace development endpoint.
You must select “Dev” when choosing the “Target sub-resource”. This will refer to the Synapse Workspace Development endpoint.
Select the Virtual Network where you have hosted the DevOps Self-hosted agent VM
After the “Review and create” step, hit the “Create” button to create the Synapse Workspace Private Endpoint.
If you check your Virtual Network Private Endpoints you’ll see the newly create private endpoint:
Check your Private Endpoint DNS Configuration and look for the IP Address and FQDN
If you "nslookup" the Synapse Workspace from your VM, confirm if values from the output match the Private DNS Config:
Before you kick-off the DevOps Release Pipeline, you need also to confirm that the DevOps Connection Service Principal that is being used by Azure Synapse Deployment task has been granted the necessary Synapse RBAC permissions (if not, the task will fail with a 403 authorization error)
Finally, you can create your first release to start deploying your Synapse Workspace artifacts to the target Managed VNET Synapse Workspace:
You should now see the Synapse Workspace Deployment task succeeded!
A final note to thank to my colleagues @silasmendes and @FonsecaSergio for all the support
